Evaluating the morphological competence of Machine Translation Systems

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 43–55 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Evaluating the morphological competence of Machine Translation Systems Franck Burlot and Franc¸ois Yvon LIMSI, CNRS, Universit´e Paris-Saclay, France firstname.lastname@limsi.fr Abstract While recent changes in Machine Trans- lation state-of-the-art brought translation quality a step further, it is regularly ac- knowledged that the standard automatic metrics do not provide enough insights to fully measure the impact of neural models. This paper proposes a new type of evalu- ation focused specifically on the morpho- logical competence of a system with re- spect to various grammatical phenomena. Our approach uses automatically gener- ated pairs of source sentences, where each pair tests one morphological contrast. This methodology is used to compare several systems submitted at WMT’17 for English into Czech and Latvian. 1 Introduction It is nowadays unanimously recognized that Ma- chine Translation (MT) engines based on the neu- ral encoder-decoder architecture with attention (Cho et al., 2014; Bahdanau et al., 2014) constitute the new state-of-the-art in statistical MT, at least for open-domain tasks (Sennrich et al., 2016a). The previous phrase-based (PBMT) architectures were complex (Koehn, 2010) and hard to diag- nose, and Neural MT (NMT) systems, which dis- pense with any sort of symbolic representation of the learned knowledge, are probably worse in this respect. Furthermore, the steady progress of MT engines makes automatic metrics such as BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005) less appropriate to evaluate and com- pare modern NMT systems. To better understand the strength and weaknesses of these new architec- tures, it is thus necessary to investigate new, more focused, evaluation procedures. Error analysis protocols, as proposed eg. by Vilar et al. (2006); Popovi´c and Ney (2011) for PBMT, are obvious candidates for such studies and have been used eg. in (Bentivogli et al., 2016). Recently, various new proposals have been put for- ward to better diagnose neural models, notably by Linzen et al. (2016); Sennrich (2017), who focus respectively on the syntactic competence of Neu- ral Language Models (NLMs) or of NMT; and by Isabelle et al. (2017); Burchardt et al. (2017), who resuscitate an old tradition of designing test suites. Inspired by these (and other) works (see § 4), we propose in this paper a new evaluation scheme aimed at specifically assessing the morphologi- cal competence of MT engines translating from English into a Morphologically Rich Language (MRL). Morphology poses two main types of problems in MT: (a) morphological variation in the source increases the occurrence of Out-of- Vocabulary (OOV) source tokens, the translation of which is difficult to coin; (b) morphological variation in the target forces the MT to generate forms that have not been seen in training. Morpho- logical complexity is alo often associated to more flexible word orderings, which is mostly a prob- lem when translating from a MRL (Bisazza and Federico, 2016). Reducing these issues is a legiti- mate and important goal for many language …

Content truncated to 500 words