Non-Autoregressive Neural Machine Translation with Consistency Regularization Optimized Variational Framework

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 607 - 617 July 10-15, 2022 ©2022 Association for Computational Linguistics Non-Autoregressive Neural Machine Translation with Consistency Regularization Optimized Variational Framework Minghao Zhu and Junli Wang and Chungang Yan Key Lab of Embedded System and Service Computing (Tongji University), Ministry of Education, Shanghai 201804, China. National (Province-Ministry Joint) Collaborative Innovation Center for Financial Network Security, Tongji University, Shanghai 201804, China. {mzhu, junliwang, yanchungang}@tongji.edu.cn Abstract Variational Autoencoder (VAE) is an effective framework to model the interdependency for non-autoregressive neural machine translation (NAT). One of the prominent VAE-based NAT frameworks, LaNMT, achieves great improve- ments to vanilla models, but still suffers from two main issues which lower down the transla- tion quality: (1) mismatch between training and inference circumstances and (2) inadequacy of latent representations. In this work, we target on addressing these issues by proposing pos- terior consistency regularization. Specifically, we first perform stochastic data augmentation on the input samples to better adapt the model for inference circumstance, and then conduct consistency training on posterior latent vari- ables to construct a more robust latent repre- sentations without any expansion on latent size. Experiments on En<->De and En<->Ro bench- marks confirm the effectiveness of our methods with about 1.5/0.7 and 0.8/0.3 BLEU points improvement to the baseline model with about 12.6× faster than autoregressive Transformer. 1 Introduction Neural Machine Translation (NMT) achieves great success in recent years, and typical sequence-to- sequence frameworks like Transformer (Vaswani et al., 2017) achieved state-of-the-art performance on the task of NMT. In this framework, source sentences are translated in an autoregressive (AT) manner where each token is generated depend- ing on previously generated tokens. Inevitably, such sequential decoding strategy results in a high inference latency. To alleviate this issue, Non- autoregressive translation (NAT; Gu et al., 2018) was proposed to speed-up decoding procedure by generating target tokens in parallel. However, the translation quality of vanilla NAT is compro- mised, one of the most significant problems is multi-modality and it usually results in multiple translation results, duplicate or missing words in target sentences of NAT models (Gu et al., 2018). This situation results from the conditional indepen- dence proposed by NAT, since models are trained to maximize the log-probability of target tokens at each position while the interdependency is omitted. The key to alleviate the multi-modality issue is to model the dependency information of targets implicitly or explicitly so decoder can easily learn and capture the information between target tokens and generate more accurate translations. For exam- ple, Ghazvininejad et al. (2019) and (2020b) model the target dependency by providing observed tar- get tokens in training and performing iterative in- ference. Ran et al. (2021) generates intermediate representations by permuting the source sentences in the target order. Libovick`y and Helcl (2018), Shao et al. (2020) and Ghazvininejad et al. (2020a) model the target dependency by introducing objec- tive functions with alignment. Guo et al. (2020) and Wang et …

Content truncated to 500 words