Can Latent Alignments Improve Autoregressive Machine Translation?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2637–2641 June 6–11, 2021. ©2021 Association for Computational Linguistics 2637 Can Latent Alignments Improve Autoregressive Machine Translation? Adi Haviv∗ Lior Vassertail∗ Omer Levy The Blavatnik School of Computer Science Tel Aviv University {adi.haviv,lior.vassertail,levyomer}@cs.tau.ac.il Abstract Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models. Can they improve autoregressive models as well? We explore the possibility of training autoregressive machine translation models with latent alignment ob- jectives, and observe that, in practice, this ap- proach results in degenerate models. We pro- vide a theoretical explanation for these empiri- cal results, and prove that latent alignment ob- jectives are incompatible with teacher forcing. 1 Introduction Latent alignment objectives, such as CTC (Graves et al., 2006) and AXE (Ghazvininejad et al., 2020a), have been recently proposed for training non- autoregressive models for machine translation (Li- bovický and Helcl, 2018; Saharia et al., 2020). These objectives use a dynamic program to comb the space of monotonic alignments between the “gold” target sequence and the token probabilities the model predicts, thus reducing the loss from po- sitional misalignments and focusing on the original prediction error instead. For example, consider the target sequence “there is a tiny difference between pink and magenta”; if the model’s distribution fa- vors the paraphrase “there is a very small difference between pink and magenta”, substituting one token (“tiny”) with two (“very small”) will cause a mis- alignment, and result in a disproportionately large cross entropy loss. A latent alignment loss would match the predictions of both “very” and “small” with the target “tiny”, while aligning the rest of the sentence properly and computing a much lower loss that focuses on this particular discrepancy. Could latent alignments also benefit autoregressive mod- els? We apply CTC and AXE to standard autoregres- sive machine translation models. We observe that, ∗ Equal contribution. when trained with teacher forcing (Williams and Zipser, 1989), CTC reduces to the vanilla cross en- tropy loss because CTC assumes that the prediction sequence is longer than the target and has only one valid alignment when they are equal. We further examine AXE, which does not share this assump- tion, and find that it yields a degenerate model that almost perfectly fits the training set but completely fails at inference time. Our analysis reveals that latent alignments and teacher forcing are fundamentally incompatible. We observe that there exists a valid alignment in which the prediction pi is aligned with the target yi−1 for almost every token. Simultane- ously, teacher forcing feeds the model with yi−1 when computing the prediction pi, encouraging the model to simply predict its input under this alignment. While AXE allows this alignment for equal-length prediction and target sequences, the phenomenon also occurs (theoretically) in CTC if the predictions are longer, and in fact, occurs in any latent alignment objective that can align a prediction pj with a target yi where i < …

Content truncated to 500 words