It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1640–1649 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1640 It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information Emanuele BugliarelloC Sabrina J. MielkeH Antonios Anastasopoulos@ Ryan CotterellD,Q Naoaki OkazakiN CUniversity of Copenhagen HJohns Hopkins University @Carnegie Mellon University DUniversity of Cambridge QETH Z¨urich NTokyo Institute of Technology emanuele@di.ku.dk, sjmielke@jhu.edu, aanastas@cs.cmu.edu, rcotterell@inf.ethz.ch, okazaki@c.titech.ac.jp Abstract The performance of neural machine transla- tion systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more dif- ficult to model. In this paper, we propose cross-mutual information (XMI): an asymmet- ric information-theoretic metric of machine translation difficulty that exploits the proba- bilistic nature of most neural machine trans- lation models. XMI allows us to better eval- uate the difficulty of translating text into the target language while controlling for the dif- ficulty of the target-side generation compo- nent independent of the translation task. We then present the first systematic and con- trolled study of cross-lingual translation dif- ficulties using modern neural translation sys- tems. Code for replicating our experiments is available online at https://github.com/ e-bug/nmt-difficulty. 1 Introduction Machine translation (MT) is one of the core re- search areas in natural language processing. Cur- rent state-of-the-art MT systems are based on neu- ral networks (Sutskever et al., 2014; Bahdanau et al., 2015), which generally surpass phrase-based systems (Koehn, 2009) in a variety of domains and languages (Bentivogli et al., 2016; Toral and S´anchez-Cartagena, 2017; Castilho et al., 2017; Bojar et al., 2018; Barrault et al., 2019). Using phrase-based MT systems, various controlled stud- ies to understand where the translation difficulties lie for different language pairs were conducted (Birch et al., 2008; Koehn et al., 2009). However, comparable studies have yet to be performed for neural machine translation (NMT). As a result, it is still unclear whether all translation directions are equally easy (or hard) to model for NMT. This pa- per hence aims at filling this gap: Ceteris paribus, MI: characterize language H(S) H(S | T) MI(S ; T) ⇒ intrinsic source/target language variation shared infor- mation H(T) H(T | S) MI(S ; T) ⇐ XMI: characterize models HqLM(S) HqMT(S | T) XMI(T → S) ⇒ intrinsic source/target modeling difficulty transfer diffi- culty HqLM(T) HqMT(T | S) XMI(S → T) ⇐ Figure 1: Left: Decomposing the uncertainty of a sentence as mutual information plus language-inherent uncertainty: mutual information (MI) corresponds to just how much easier it becomes to predict T when you are given S. MI is symmetric but the relation be- tween H(S) and H(T) can be arbitrary. Right: estimat- ing cross-entropies using models qMT and qLM invali- dates relations between bars, except that Hq·(·) ≥ H(·). XMI, our proposed metric, is no longer purely a sym- metric measure of language, but now an asymmetric measure that mostly highlights …

Content truncated to 500 words