NTR organizes and hosts scientific webinars on neural networks and invites speakers from all over the world to present their recent work at the webinars.
On July 20 Tatiana Likhomanenko, Postdoctoral Researcher, Facebook AI Research, Menlo Park, California, USA, led a technical Zoom webinar on slimIPL: Language-Model-Free Iterative Pseudo-Labeling.
About the webinar:
Recent results in end-to-end automatic speech recognition have demonstrated the efficacy of pseudo-labeling for semi-supervised models trained with both Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses.
Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further improve performance in ASR.
We improved upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens), without a language model.
We call this approach Language-Model-Free IPL (slimIPL) and give a resultant training setup for low-resource settings with CTC-based models.
slimIPL features a dynamic cache for pseudo-labels that reduces sensitivity to changes in relabeling hyperparameters and results in improved training stability.
slimIPL is also highly efficient and requires 3.5-4x fewer computational resources for converge than other state-of-the-art semi/self-supervised approaches.
With only 10 hours of labeled audio, slimIPL is competitive with self-supervised approaches, and is state-of-the-art with 100 hours of labeled audio, without the use of a language model, both at test time and during pseudo-label generation.
Webinar presentation.
Moderator and contact: NTR CEO Nick Mikhailovsky: nickm@ntrlab.com.