Machine Translation Robustness to Natural Asemantic Variation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3517 - 3532 December 7-11, 2022 ©2022 Association for Computational Linguistics Machine Translation Robustness to Natural Asemantic Variation Jacob Bremerman Xiang Ren Jonathan May∗ University of Southern California Information Sciences Institute {jbrem,xiangren,jonmay}@usc.edu Abstract Current Machine Translation (MT) models still struggle with more challenging input, such as noisy data and tail-end words and phrases. Sev- eral works have addressed this robustness issue by identifying specific categories of noise and variation then tuning models to perform bet- ter on them. An important yet under-studied category involves minor variations in nuance (non-typos) that preserve meaning w.r.t. the target language. We introduce and formal- ize this category as Natural Asemantic Vari- ation (NAV) and investigate it in the context of MT robustness. We find that existing MT models fail when presented with NAV data, but we demonstrate strategies to improve per- formance on NAV by fine-tuning them with human-generated variations. We also show that NAV robustness can be transferred across lan- guages and find that synthetic perturbations can achieve some but not all of the benefits of or- ganic NAV data. 1 Introduction While current Machine Translation (MT) models often perform decently in regular conditions, ro- bustness to slight modifications on input remains a challenge. Several works have investigated the dev- astating effects that minor noise can cause in MT (Belinkov and Bisk, 2018; Khayrallah and Koehn, 2018). Addressing this is critical as robustness to naturally occurring variation is important in de- ployed systems. Some classes of perturbations to which humans exhibit robustness but studies have shown MT models struggle with include speling errors or rtypos, CaSinG (Niu et al., 2020), v15sua11y- 51m1ll4r ch4r4ct3r5 (Eger et al., 2019), and syn- onym replacement substitution (Wang et al., 2018). However, we focus on a less-known class we call Natural Asemantic Variation (NAV). ∗Work done prior to JM joining Amazon. Perturbation Asemantic Variation 彼女は私に本を返した informal verb conjugation 彼女は俺に本を返しました masculine "me" pronoun 彼女は私に本を返してくれました “favor” nuance Table 1: Examples of NAV perturbations for a Japanese sen- tence, 彼女は私に本を返しました (she returned the book to me). These natural variations, which cause nuance differences in Japanese, cannot be translated succinctly into English. Therefore, all perturbations can be translated reason- ably to the same English sentence. A robust MT model (or human) should be able to extract the same semantic meaning regardless of the variation provided. Whereas aforementioned classes tend towards perturbations that generate agrammatical, nonstan- dard, or unnatural sentences, NAV perturbations represent the extra-semantic linguistic properties of a language that allow for subtle changes in nuance while expressing the same core meaning. Specifi- cally, a NAV perturbation is a kind of paraphrase in a source language that would not affect its trans- lation in a given target language. Studying NAV is an important endeavor in MT robustness since it covers a less-contrived source of variation com- pared to existing MT robustness studies. As this topic is under-studied, it has also not been well- defined. We offer an example of …

Content truncated to 500 words