Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese

Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese Yuying Ye Digital Humanities Programme University of Groningen The Netherlands y.ye.yuying@gmail.com Antonio Toral Center for Language and Cognition University of Groningen The Netherlands a.toral.ruiz@rug.nl Abstract This research presents a fine-grained hu- man evaluation to compare the Trans- former and recurrent approaches to neural machine translation (MT), on the transla- tion direction English-to-Chinese. To this end, we develop an error taxonomy com- pliant with the Multidimensional Quality Metrics (MQM) framework that is cus- tomised to the relevant phenomena of this translation direction. We then conduct an error annotation using this customised er- ror taxonomy on the output of state-of-the- art recurrent- and Transformer-based MT systems on a subset of WMT2019’s news test set. The resulting annotation shows that, compared to the best recurrent sys- tem, the best Transformer system results in a 31% reduction of the total number of er- rors and it produced significantly less er- rors in 10 out of 22 error categories. We also note that two of the systems evaluated do not produce any error for a category that was relevant for this translation direction prior to the advent of NMT systems: Chi- nese classifiers. 1 Introduction The field of machine translation (MT) has been revolutionised in the past few years by the emer- gence of a new approach: neural MT (NMT). NMT is a dynamic research area and we have witnessed two mainstream architectures already, the first of which is based on recurrent neural c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. networks (RNN) with attention (Bahdanau et al., 2014) while the second, referred to as Transformer, makes use of the self-attention mechanism in non- recurrent networks (Vaswani et al., 2017). Several studies have analysed in depth, using both automatic and human evaluation methods, the resulting translations of NMT systems under the recurrent architecture and compared them to the translations of the previous mainstream approach to MT: statistical MT (Koehn et al., 2003), e.g. (Bentivogli et al., 2016; Castilho et al., 2017; Klubiˇcka et al., 2018; Popovi´c, 2017; Shterionov et al., 2018). However, while the Transformer ar- chitecture has brought, at least when trained with sufficient data, considerable gains over the recur- rent architecture (Vaswani et al., 2017), the re- search conducted to date that analyses the result- ing translations of these two neural approaches is, to the best of our knowledge, limited to automatic approaches (Burlot et al., 2018; Lakew et al., 2018; Tang et al., 2018a; Tang et al., 2018b; Tran et al., 2018; Yang et al., 2019). In this paper we conduct a detailed human anal- ysis of the outputs produced by state-of-the-art re- current and Transformer NMT systems. Namely, we manually annotate the errors found according to a detailed error taxonomy which is compliant with the hierarchical listing of issue types defined as part of the Multidimensional Quality Metrics (MQM) framework (Lommel et al., 2014). We …

Content truncated to 500 words