Categorizing Semantic Representations for Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5227–5239 October 12–17, 2022. 5227 Categorizing Semantic Representations for Neural Machine Translation Yongjing Yin1,2∗, Yafu Li1,2, Fandong Meng3, Jie Zhou3, Yue Zhang2,4 1 Zhejiang University 2 School of Engineering, Westlake University 3 Pattern Recognition Center, WeChat AI, Tencent Inc 4 Institute of Advanced Technology, Westlake Institute for Advanced Study {yinyongjing,liyafu}@westlake.edu.cn {fandongmeng,withtomzhou}@tencent.com yue.zhang@wias.org.cn Abstract Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks. However, they have recently been shown to suffer limitation in com- positional generalization, failing to effectively learn the translation of atoms (e.g., words) and their semantic composition (e.g., modification) from seen compounds (e.g., phrases), and thus suffering from significantly weakened transla- tion performance on unseen compounds dur- ing inference. We address this issue by intro- ducing categorization to the source contextu- alized representations. The main idea is to en- hance generalization by reducing sparsity and overfitting, which is achieved by finding pro- totypes of token representations over the train- ing set and integrating their embeddings into the source encoding. Experiments on a dedi- cated MT dataset (i.e., CoGnition) show that our method reduces compositional generaliza- tion error rates by 24% error reduction. In ad- dition, our conceptually simple method gives consistently better results than the Transformer baseline on a range of general MT datasets. 1 Introduction Neural machine translation (NMT) has achieved competitive performance on benchmark datasets such as WMT (Vaswani et al., 2017; Edunov et al., 2018; So et al., 2019). However, the generalizaiton to low-resource domains (Bapna and Firat, 2019b; Zeng et al., 2019; Bapna and Firat, 2019a; Khan- delwal et al., 2021) and robustness to slight input perturbations (Belinkov and Bisk, 2018; Xu et al., 2021b) are relatively low for NMT models. In addi- tion, recent studies show that NMT systems are vul- nerable to compositional generalization (Lake and Baroni, 2018; Raunak et al., 2019; Guo et al., 2020; Li et al., 2021; Dankers et al., 2021; Chaabouni et al., 2021), namely the ability to understand and ∗This work was done as an intern at Pattern Recognition Center, WeChat AI, Tencent Inc, China. …… Novel Compounds DT ADJ N the large chair the small car … … … Source Sentences with Novel Compounds The park is near the small car. …… For Pattern 1.2: Compound Patterns Pattern 1.1: DET+N Pattern 1.2: DET+ADJ+N Pattern 1.3: DET+N+MOD …… Atoms DET: the, a, any… ADJ: small, large, red… N: car, chair, doctor… …… Figure 1: The novel compounds in the CoGnition test set are constructed by composing a few basis semantic atoms (e.g., determiners (DET), nouns (N), and adjec- tives (ADJ)) according to the composition patterns. The compounds are then put into corresponding source con- texts extracted from the training data. produce a potentially infinite (formally exponential to the input size) number of novel combinations of known atoms (Chomsky, 2009; Montague and Thomason, 1975; Janssen and Partee, 1997; Lake and Baroni, 2018; Keysers et al., 2020a). Take CoGnition (Li et al., 2021), a dedicated …

Content truncated to 500 words