Evaluating Discourse Phenomena in Neural Machine Translation

Proceedings of NAACL-HLT 2018, pages 1304–1313 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Evaluating Discourse Phenomena in Neural Machine Translation Rachel Bawden1 Rico Sennrich2,3 Alexandra Birch2 Barry Haddow2 1LIMSI, CNRS, Univ. Paris-Sud, Universit´e Paris-Saclay, F-91405 Orsay, France 2School of Informatics, University of Edinburgh, Scotland 3Institute of Computational Linguistics, University of Zurich, Switzerland rachel.bawden@limsi.fr {rico.sennrich, a.birch}@ed.ac.uk bhaddow@inf.ed.ac.uk Abstract For machine translation to tackle discourse phenomena, models must have access to extra- sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard au- tomatic metrics, poorly adapted to evaluat- ing discourse phenomena. In this article, we present hand-crafted, discourse test sets, de- signed to test the models’ ability to exploit previous source and target sentences. We in- vestigate the performance of recently proposed multi-encoder NMT models trained on subti- tles for English to French. We also explore a novel way of exploiting context from the pre- vious sentence. Despite gains using BLEU, multi-encoder models give limited improve- ment in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi- encoding and decoding of two sentences leads to the best performance (72.5% for corefer- ence and 57% for coherence/cohesion), high- lighting the importance of target-side context. 1 Introduction Machine translation (MT) systems typically trans- late sentences independently of each other. How- ever, certain textual elements cannot be correctly translated without linguistic context, which may appear outside the current sentence. The most obvious examples of context-dependent phenom- ena problematic for MT are coreference (Guillou, 2016), lexical cohesion (Carpuat, 2009) and lex- ical disambiguation (Rios Gonzales et al., 2017), an example for each of which is given in (1-3). In each case, the English element in italic is ambigu- ous in terms of its French translation. The correct translation choice (in bold) is determined by lin- guistic context (underlined), which can be outside the current sentence. This disambiguating context can be source or target-side; the correct translation of anaphoric pronouns it and they depends on the gender of the translated antecedent (1). In lexical cohesion, a translation may depend on target fac- tors, but may also be triggered by source effects and linguistic mechanisms such as repetition or alignment (2). In lexical disambiguation, source or target information may provide the appropriate context (3). (1) The bee is busy. // It is making honey. L’abeille[f] est occup´ee. // Elle[f]/#il[m] fait du miel. (2) Do you fancy some soup? // Some soup? Tu veux de la soupe? // De la soupe/#du potage? (3) And the code? // Still some bugs... Et le code ? // Encore quelques bugs/#insectes... Recent work on multi-encoder neural machine translation (NMT) appears promising for the in- tegration of linguistic context (Zoph and Knight, 2016; Libovick´y and …

Content truncated to 500 words