ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5035–5045 October 12–17, 2022. 5035 ngram-OAXE : Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation Cunxiao Du Singapore Management University cnsdunm@gmail.com Zhaopeng Tu∗ Tencent AI Lab zptu@tencent.com Longyue Wang Tencent AI Lab vinnylywang@tencent.com Jing Jiang Singapore Management University jingjiang@smu.edu.sg Abstract Recently, a new training OAXE loss (Du et al., 2021) has proven effective to ameliorate the effect of multimodality for non-autoregressive translation (NAT), which removes the penalty of word order errors in the standard cross- entropy loss. Starting from the intuition that re- ordering generally occurs between phrases, we extend OAXE by only allowing reordering be- tween ngram phrases and still requiring a strict match of word order within the phrases. Exten- sive experiments on NAT benchmarks across language pairs and data scales demonstrate the effectiveness and universality of our approach. Further analyses show that ngram-OAXE in- deed improves the translation of ngram phrases, and produces more fluent translation with a bet- ter modeling of sentence structure.1 1 Introduction Fully non-autoregressive translation (NAT) has re- ceived increasing attention for its efficient decod- ing by predicting every target token in parallel (Gu et al., 2018; Ghazvininejad et al., 2019). However, such advantage comes at the cost of sacrificing translation quality due to the multimodality prob- lem: there exist many possible translations of the same sentence, while vanilla NAT models may con- sider them at the same time due to the independent predictions, which leads to multi-modal outputs in the form of token repetitions (Gu et al., 2018). Recent works have incorporated approaches to improving the standard cross-entropy (XE) loss to ameliorate the effect of multimodality. The motiva- tion for these works is that modeling word order is difficult for NAT, since the model cannot condition on its previous predictions like its autoregressive counterpart. Starting from this intuition, a thread of research relaxes the word order restriction based on ∗Zhaopeng Tu is the corresponding author. 1The codes and models are in https://github. com/tencent-ailab/machine-translation/ COLING22_ngram-OAXE/. the monotonic alignment assumption (Libovický and Helcl, 2018; Ghazvininejad et al., 2020; Sa- haria et al., 2020). Du et al. (2021) take a further step by removing the penalty of word order errors with a novel order-agnostic cross entropy (OAXE) loss, which enables NAT models to handle word reordering – a common source of multimodality problem. Accordingly, OAXE achieves the best performance among these model variants. However, OAXE allows reordering between ev- ery two words, which is not always valid in prac- tice. For example, the reordering of the two words “this afternoon” is not correct in grammar. The re- ordering generally occurs between ngram phrases, such as “I ate pizza” and “this afternoon”. Starting from this intuition, we extend OAXE by constrain- ing the reordering between ngrams and requiring a strict match of word order within each ngram (i.e., ngram-OAXE). To this end, we first build the probability distributions of ngrams in the target sentence using the word probabilities produced by NAT models. Then we find the …

Content truncated to 500 words