Neural versus Phrase-Based Machine Translation Quality: a Case Study

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257–267, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Neural versus Phrase-Based Machine Translation Quality: a Case Study Luisa Bentivogli FBK, Trento Italy Arianna Bisazza University of Amsterdam The Netherlands Mauro Cettolo FBK, Trento Italy Marcello Federico FBK, Trento Italy Abstract Within the field of Statistical Machine Trans- lation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particu- lar, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of- the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differ- ences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural vs. phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models – such as the reordering of verbs – while pointing out other aspects that remain to be improved. 1 Introduction The wave of neural models has eventually reached the field of Statistical Machine Translation (SMT). After a period in which Neural MT (NMT) was too computationally costly and resource demanding to compete with state-of-the-art Phrase-Based MT (PBMT)1, the situation changed in 2015. For the first time, in the latest edition of IWSLT2 (Cettolo et 1We use the generic term phrase-based MT to cover standard phrase-based, hierarchical and syntax-based SMT approaches. 2International Workshop on Spoken Language Translation (http://workshop2015.iwslt.org/) al., 2015), the system described in (Luong and Man- ning, 2015) overtook a variety of PBMT approaches with a large margin (+5.3 BLEU points) on a diffi- cult language pair like English-German – anticipat- ing what, most likely, will be the new NMT era. This impressive improvement follows the dis- tance reduction previously observed in the WMT 2015 shared translation task (Bojar et al., 2015). Just few months earlier, the NMT systems de- scribed in (Jean et al., 2015b) ranked on par with the best phrase-based models on a couple of lan- guage pairs. Such rapid progress stems from the im- provement of the recurrent neural network encoder- decoder model, originally proposed in (Sutskever et al., 2014; Cho et al., 2014b), with the use of the at- tention mechanism (Bahdanau et al., 2015). This evolution has several implications. On one side, NMT represents a simplification with respect to pre- vious paradigms. From a management point of view, similar to PBMT, it allows for a more efficient use of human and data resources with respect to rule- based MT. From the architectural point of view, a large recurrent network trained for end-to-end trans- lation is considerably simpler than traditional MT systems that integrate multiple components and pro- cessing steps. On the other side, the NMT pro- cess is less transparent than previous paradigms. In- deed, it represents a further step …

Content truncated to 500 words