Fine-grained linguistic evaluation for state-of-the-art Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 346–356 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 346 Fine-grained linguistic evaluation for state-of-the-art Machine Translation Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, Aljoscha Burchardt and Sebastian M¨oller German Research Center for Artificial Intelligence (DFKI), Berlin, Germany firstname.lastname@dfki.de Abstract This paper describes a test suite submissi- on providing detailed statistics of linguistic performance for the state-of-the-art German- English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 phenomena organized in 14 catego- ries based on about 5,500 test items, including a manual annotation effort of 45 person hours. Two systems (Tohoku and VolcanTrans) appe- ar to have significantly better test suite accu- racy than the others, although the best system of WMT20 is not significantly better than the one from WMT19 in a macro-average. Addi- tionally, we identify some linguistic phenome- na where all systems suffer (such as idioms, re- sultative predicates and pluperfect), but we are also able to identify particular weaknesses for individual systems (such as quotation marks, lexical ambiguity and sluicing). Most of the systems of WMT19 which submitted new ver- sions this year show improvements. 1 Introduction Fine-grained evaluation has recently had increa- sing interest on several natural language processing (NLP) tasks. Focusing on particular issues gives the possibility to analyse the automatic output in ways that cannot be seen by generic metrics. This is of particular importance in the era of deep learning, which has led to high performances and differences that are relatively difficult to distinguish. Additio- nally, detailed evaluation can provide indications for the improvement of the systems and the da- ta collection, or allow focusing on phenomena of the long tail that might be of particular interest for certain cases (e.g. social biases; Stanovsky et al., 2019). The most common method for fine-grained or fo- cused evaluation are the test suites (also known as challenge sets or benchmarks; Guillou and Hard- meier, 2016; Ribeiro et al., 2020). These are test sets engineered in a particular way, so that they can test the performance of NLP tasks on concrete issues (M¨uller et al., 2018; Bawden et al., 2018). This paper is presenting the use of such a test suite for the evaluation of the 11 German→English Machine Translation (MT) systems that participa- ted at the Shared Task of the Fifth Conference of Machine Translation (WMT20; Barrault et al., 2020). The evaluation applies the DFKI test sui- te on German-English, via 5,514 test items which cover 107 linguistically motivated phenomena orga- nized in 14 categories. After a reference in related work (Section 2), we explain shortly the structure of the test suite (Section 3) and present the results (Section 4) and the conclusions (Section 5). 2 Related Work The use of test suites was introduced along with the early steps of MT in the 1990’s (King and Fal- kedal, 1990; Way, 1991; Heid and Hildenbrand, 1991). With the emergence of deep learning, re- cent works re-introduced test suites that …

Content truncated to 500 words