Revisiting Negation in Neural Machine Translation

Revisiting Negation in Neural Machine Translation Gongbo Tang1 Philipp R¨onchen1 Rico Sennrich2,3 Joakim Nivre1 1Department of Linguistics and Philology, Uppsala University, Sweden 2Department of Computational Linguistics, University of Zurich, Switzerland 3School of Informatics, University of Edinburgh gongbo.tang,philipp.r¨onchen,joakim.nivre{@lingfil.uu.se} rico.sennrich@ed.ac.uk Abstract In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN→DE, DE→EN, EN→ZH, and ZH→EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under- translation of negation, we study the model’s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement. 1 Introduction Negation is an important linguistic phenomenon in machine translation, as errors in translating nega- tion may change the meaning of source sentences completely. There are many studies on negation in statistical machine translation (SMT) (Collins et al., 2005; Li et al., 2009; Wetzel and Bond, 2012; Baker et al., 2012; Fancellu and Webber, 2014, 2015), but studies on negation in neu- ral machine translation (NMT) are quite limited and results are partly conflicting. For example, Bentivogli et al. (2016) find that negation is still challenging, whereas Bojar et al. (2018) show that NMT models almost make no mistakes on negation using 130 sentences with negation from three language pairs as the evaluation set. Hence, it is still not clear how well NMT models perform on the translation of negation. In this paper, we present both automatic and manual evaluation of negation in NMT, in English–German (EN–DE) and English–Chinese (EN–ZH). The automatic evaluation is based on contrastive translation pairs and studies transla- tion from English into German/Chinese (EN→ DE/ZH). The manual evaluation targets transla- tion in all four translation directions. We find that the modeling of negation in NMT has improved with deeper and more advanced networks. The contrastive evaluation shows that deleting nega- tion from references is more confusing to NMT models compared to inserting negation into ref- erences. For the manual evaluation, NMT models make fewer mistakes on negation in EN–DE, than in EN–ZH, and there are more errors on negation in DE/ZH→EN than in EN→DE/ZH. Moreover, under-translation is the …

Content truncated to 500 words