A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1063–1073, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions Antonio Toral∗ University of Groningen The Netherlands a.toral.ruiz@rug.nl V´ıctor M. S´anchez-Cartagena Prompsit Language Engineering Av. Universitat s/n. Edifici Quorum III E-03202 Elx, Spain vmsanchez@prompsit.com Abstract We aim to shed light on the strengths and weaknesses of the newly introduced neu- ral machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation sys- tems for 9 language directions across a number of dimensions. Specifically, we measure the similarity of the outputs, their fluency and amount of reordering, the ef- fect of sentence length and performance across different error categories. We find out that translations produced by neural machine translation systems are consider- ably different, more fluent and more ac- curate in terms of word order compared to those produced by phrase-based sys- tems. Neural machine translation sys- tems are also more accurate at producing inflected forms, but they perform poorly when translating very long sentences. 1 Introduction A new paradigm to statistical machine transla- tion, neural MT (NMT), has emerged very re- cently and has already surpassed the performance of the mainstream approach in the field, phrase- based MT (PBMT), for a number of language pairs, e.g. (Sennrich et al., 2016b; Luong et al., 2015; Costa-Juss`a and Fonollosa, 2016; Chung et al., 2016). In PBMT (Koehn, 2010) different models (translation, reordering, target language, etc.) are trained independently and combined in a log- linear scheme in which each model is assigned a ∗Work partly done at his previous position in Dublin City University, Ireland. different weight by a tuning algorithm. On the contrary, in NMT all the components are jointly trained to maximise translation quality. NMT sys- tems have a strong generalisation power because they encode translation units as numeric vectors that represent concepts, whereas in PBMT transla- tion units are encoded as strings. Moreover, NMT systems are able to model long-distance phenom- ena thanks to the use of recurrent neural networks, e.g. long short-term memory (LSTM) (Hochre- iter and Schmidhuber, 1997) or gated recurrent units (Chung et al., 2014). The translations produced by NMT systems have been evaluated thus far mostly in terms of overall performance scores, be it by means of au- tomatic or human evaluations. This has been the case of last year’s news translation shared task at the First Conference on Machine Translation (WMT16).1 In this translation task, outputs pro- duced by participant MT systems, the vast ma- jority of which fall under either the phrase-based or neural approaches, were evaluated (i) automat- ically with the BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) metrics, and (ii) manu- ally by means of ranking translations (Federmann, 2012) and monolingual semantic similarity (Gra- ham et al., 2016). …

Content truncated to 500 words