On Compositional Generalization of Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4767–4780 August 1–6, 2021. ©2021 Association for Computational Linguistics 4767 On Compositional Generalization of Neural Machine Translation Yafu Li♠~ , Yongjing Yin♠~ , Yulong Chen♠~ , Yue Zhang~} ♠ Zhejiang University ~ School of Engineering, Westlake University } Institute of Advanced Technology, Westlake Institute for Advanced Study yafuly@gmail.com yinyongjing@westlake.edu.cn yulongchen1010@gmail.com yue.zhang@wias.org.cn Abstract Modern neural machine translation (NMT) models have achieved competitive perfor- mance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, con- sisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of var- ious factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, al- though it performs remarkably well under tra- ditional metrics. 1 Introduction Neural machine translation (NMT) has shown com- petitive performance on benchmark datasets such as IWSLT and WMT (Vaswani et al., 2017; Edunov et al., 2018a; Liu et al., 2020a), and even achieves parity with professional human translation under certain evaluation settings (Hassan et al., 2018). However, the performance can be relatively low in out-of-domain and low-resource conditions. In addition, NMT systems show poor robustness and vulnerability to input perturbations (Belinkov and Bisk, 2018a; Cheng et al., 2019). One example is shown in Table 1, where simple substitution of a word yields translation with completely different semantics. Many of these issues origin from the fact that NMT models are trained end-to-end over large parallel data, where new test sentences can be sparse. Disregarding out-of-vocabulary words, a main cause of sparsity is semantic composition: given a limited vocabulary, the number of possible com- positions grows exponentially with respect to the composite length. The ability to understand and Input Translation Taylor breaks his promise “·à˙言 (Taylor keeps his promise) James breaks his promise y∆Ø›Õ˙言 (James breaks his promise) Table 1: Translation samples obtained from one popu- lar web translation engine on January 19, 2021. produce a potentially infinite number of novel com- binations of known components, namely composi- tional generalization (Chomsky; Montague; Lake and Baroni, 2018; Keysers et al., 2020), has been demonstrated deficient in many machine learning (ML) methods (Johnson et al., 2017a; Lake and Baroni, 2018; Bastings et al., 2018; Loula et al., 2018; Russin et al., 2019a). In this paper, we study compositional general- ization in the context of machine translation. For example, if “red cars” and “blue balls” are seen in training, a competent algorithm is expected to translate “red balls” correctly, even if the phrase has not been seen in training data. Intuitively, the challenge increases as the composite length grows. Recently, several studies have taken steps towards this specific problem. They either use a few dedi- cated samples (i.e., 8 test sentences) for evaluation (Lake and Baroni, 2018; Li …

Content truncated to 500 words