Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1920–1927 June 6–11, 2021. ©2021 Association for Computational Linguistics 1920 Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment Ethan A. Chi Stanford University∗ ethanchi@cs.stanford.edu Julian Salazar Amazon AWS AI julsal@amazon.com Katrin Kirchhoff Amazon AWS AI katrinki@amazon.com Abstract Non-autoregressive encoder-decoder models greatly improve decoding speed over autore- gressive models, at the expense of genera- tion quality. To mitigate this, iterative decod- ing models repeatedly infill or refine the pro- posal of a non-autoregressive model. How- ever, editing at the level of output sequences limits model flexibility. We instead pro- pose iterative realignment, which by refin- ing latent alignments allows more flexible edits in fewer steps. Our model, Align- Refine, is an end-to-end Transformer which it- eratively realigns connectionist temporal clas- sification (CTC) alignments. On the WSJ dataset, Align-Refine matches an autoregres- sive baseline with a 14× decoding speedup; on LibriSpeech, we reach an LM-free test- other WER of 9.0% (19% relative improve- ment on comparable work) in three iterations. We release our code at https://github.com/ amazon-research/align-refine. 1 Introduction Transformer encoder-decoder models (Vaswani et al., 2017) have achieved high performance in sequence-to-sequence tasks like neural machine translation (NMT; Edunov et al., 2018) and end- to-end automatic speech recognition (ASR; Karita et al., 2019). However, like their recurrent predeces- sors, these models are autoregressive at inference: tokens are generated sequentially, with each token conditioned on all previous tokens. This makes decoding time linear in output sequence length, which is slow for long sequences. By contrast, non- autoregressive (NAR) models decode all output tokens independently and in parallel. When com- bined with self-attention, one gets fast, constant- time inference in NMT (Gu et al., 2018) and end- to-end ASR (Salazar et al., 2019). However, these models underperform their autoregressive counter- ∗Work done during an internship at Amazon AWS AI. Figure 1: In Align-Refine, a Transformer encoder la- bels each input frame to give a latent alignment. The re- finer, a non-causal Transformer decoder, improves the alignment conditioned on the encoder. After a bounded # of iterations, the result is collapsed into the output. parts, as the conditional independence between out- put tokens results in globally inconsistent outputs.1 To mitigate these issues, infilling methods like Mask-Predict (Ghazvininejad et al., 2019) refine an initial non-autoregressive proposal, repeatedly predicting a masked subset of low-confidence pro- posal tokens in a fixed number of decoding passes. In ASR, most iterative non-autoregressive methods use infilling, like A-FMLM (Chen et al., 2020), Im- puter (Chan et al., 2020), and Mask-CTC (Higuchi et al., 2020). However, during training, infilling requires partial proposals to be simulated by syn- thetically masking ground truths or samples from an expert. The resulting train-test mismatch leads to poor-quality generation. An alternative proposed in NMT is iterative refinement (Lee et al., 2018); here, full proposals are predicted and trained on at each iteration, with no masking required. This 1Gu et al. call this the multimodality problem, as it is …

Content truncated to 500 words