It’s not a Non-Issue: Negation as a Source of Error in Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3869–3885 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3869 It’s not a Non-Issue: Negation as a Source of Error in Machine Translation Md Mosharaf Hossain,8 Antonios Anastasopoulos,Z Eduardo Blanco,8 and Alexis PalmerP 8Department of Computer Science and Engineering, University of North Texas ZDepartment of Computer Science, George Mason University PDepartment of Linguistics, University of North Texas mdmosharafhossain@my.unt.edu {eduardo.blanco,alexis.palmer}@unt.edu antonis@gmu.edu Abstract As machine translation (MT) systems progress at a rapid pace, questions of their adequacy linger. In this study we focus on negation, a universal, core property of human language that significantly affects the semantics of an utter- ance. We investigate whether translating nega- tion is an issue for modern MT systems using 17 translation directions as test bed. Through thorough analysis, we find that indeed the pres- ence of negation can significantly impact down- stream quality, in some cases resulting in qual- ity reductions of more than 60%. We also pro- vide a linguistically motivated analysis that di- rectly explains the majority of our findings. We release our annotations and code to replicate our analysis here: https://github.com/ mosharafhossain/negation-mt. 1 Introduction Machine translation (MT) has come a long way, steadily improving the quality of automatic sys- tems, relying mostly on the advent of neural tech- niques (Goldberg, 2017) and availability of large data collections. As neural MT (NMT) systems start to become ubiquitous, however, one should take note of whether the improvements, as mea- sured by evaluation campaigns such as WMT (Bo- jar et al., 2017, 2018a, 2019, et alia) or IWSLT shared tasks (Cettolo et al., 2017; Niehues et al., 2018, et alia) or other established benchmarks, do in fact mean better quality, especially with regards to the semantic adequacy of translations. We focus specifically on negation, a single yet quite complex phenomenon, which can severely impact the semantic content of an utterance. As NMT systems increasingly improve, to the point of claiming human parity in some high-resource lan- guage pairs and in-domain settings (Hassan et al., 2018),1 fine-grained semantic differences become 1We direct the reader to (Läubli et al., 2018) and (Toral increasingly important. Negation in particular, with its property of logical reversal, has the potential to cause loss of (or mis-)information if mistranslated. Other linguistic phenomena and analysis axes have gathered significant attention in NMT evalua- tion studies, including anaphora resolution (Hard- meier et al., 2014; Voita et al., 2018) and pronoun translation (Guillou and Hardmeier, 2016), modal- ity (Baker et al., 2012), ellipsis and deixis (Voita et al., 2019), word sense disambiguation (Tang et al., 2018), and morphological competence (Bur- lot and Yvon, 2017). Nevertheless, the last com- prehensive study of the effect of negation in MT pertains to older, phrase-based models (Fancellu and Webber, 2015). In this work, we set out to study the effect of negation in modern NMT sys- tems. Specifically, we explore: 1. Whether negation affects the quality of the produced translations (it does); 2. Whether the typically-used evaluation datasets …

Content truncated to 500 words