A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1747 - 1757 July 10-15, 2022 ©2022 Association for Computational Linguistics A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation Kexun Zhang1∗, Rui Wang2†, Xu Tan2, Junliang Guo2, Yi Ren1, Tao Qin2, Tie-Yan Liu2 1Zhejiang University, 2Microsoft Research Asia 1{kexunz,rayeren}@zju.edu.cn 2{ruiwa,xuta,junliangguo,taoqin,tyliu}@microsoft.com Abstract It is difficult for non-autoregressive translation (NAT) models to capture the multi-modal dis- tribution of target translations due to their con- ditional independence assumption, which is known as the “multi-modality problem”, in- cluding the lexical multi-modality and the syn- tactic multi-modality. While the first one has been well studied, the syntactic multi-modality brings severe challenge to the standard cross entropy (XE) loss in NAT and is under studied. In this paper, we conduct a systematic study on the syntactic multi-modality problem. Specif- ically, we decompose it into short- and long- range syntactic multi-modalities and evaluate several recent NAT algorithms with advanced loss functions on both carefully designed syn- thesized datasets and real datasets. We find that the Connectionist Temporal Classification (CTC) loss and the Order-Agnostic Cross En- tropy (OAXE) loss can better handle short- and long-range syntactic multi-modalities respec- tively. Furthermore, we take the best of both and design a new loss function to better han- dle the complicated syntactic multi-modality in real-world datasets. To facilitate practical usage, we provide a guide to use different loss functions for different kinds of syntactic multi- modality. 1 Introduction Traditional Neural Machine Translation (NMT) models predict each target token conditioned on previous generated tokens in an autoregressive way (Vaswani et al., 2017), resulting in high la- tency in inference. Non-Autoregressive Transla- tion (NAT) models generate all the target tokens in parallel (Gu et al., 2018), significantly reduc- ing inference latency. A disadvantage of NAT is that it suffers from the multi-modality problem (Gu et al., 2018) when a source sentence corresponds to multiple correct translations (Ott et al., 2018). ∗This work was conducted at Microsoft Research Asia. †Corresponding author. There are two types of multi-modalities: the lex- ical one and the syntactic one. The former one has been adequately studied (Gu et al., 2018; Zhou et al., 2020; Ding et al., 2021), while the latter one brings severe challenges to the widely used cross entropy (XE) loss in NAT. With standard XE loss, the generated tokens are required to be strictly aligned with ground truth tokens in the same posi- tions, which fails to provide positive feedback for correctly predicted words on different positions as shown in Fig. 1a. Therefore, advanced loss func- tions are introduced to provide better feedback for NAT training: Connectionist Temporal Classifica- tion (CTC) loss (Libovický and Helcl, 2018) con- siders all possible monotonic alignments between a generated sequence and the ground truth; Aligned Cross-Entropy (AXE) loss (Ghazvininejad et al., 2020) selects the best monotonic alignment; and Order-Agnostic cross entropy (OAXE) loss (Du et al., 2021) calculates the XE loss with the best alignment based on maximum …

Content truncated to 500 words