Linguistic Evaluation for the 2021 State-of-the-art Machine Translation Systems for German to English and English to German

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1059–1073 November 10–11, 2021. ©2021 Association for Computational Linguistics 1059 Linguistic evaluation for the 2021 state-of-the-art Machine Translation systems for German to English and English to German Vivien Macketanz, Eleftherios Avramidis, Shushen Manakhimova, Sebastian Möller German Research Center for Artificial Intelligence (DFKI), Berlin, Germany firstname.lastname@dfki.de Abstract We are using a semi-automated test suite in order to provide a fine-grained linguistic eval- uation for state-of-the-art machine translation systems. The evaluation includes 18 German to English and 18 English to German systems, submitted to the Translation Shared Task of the 2021 Conference on Machine Translation. Our submission adds up to the submissions of the previous years by creating and applying a wide-range test suite for English to German as a new language pair. The fine-grained eval- uation allows spotting significant differences between systems that cannot be distinguished by the direct assessment of the human evalua- tion campaign. We find that most of the sys- tems achieve good accuracies in the majority of linguistic phenomena but there are few phe- nomena with lower accuracy, such as the id- ioms, the modal pluperfect and the German resultative predicates. Two systems have sig- nificantly better test suite accuracy in macro- average in every language direction, Online-W and Facebook-AI for German to English and VolcTrans and Online-W for English to Ger- man. The systems show a steady improvement as compared to previous years. 1 Introduction Evaluation in NLP and particularly in Machine Translation (MT) is an essential process for iden- tifying flaws and leading further system improve- ments. Nevertheless, the exact method of evalu- ation to be used varies, given the quality require- ments of the particular use case. Whereas the vast majority of the evaluation methods reside on met- rics or direct assessment by humans to produce a single quality score given an entire test set, a re- cent trend has opted to evaluating the details of the produced translations, with major focus on their correctness from a linguistic perspective. For this reason, the translation systems are not tested based on generic test-sets, but they are given input which is particularly crafted to trial their performance. Most commonly, this is done with the help of a test suite (cf. Guillou and Hardmeier, 2016; Isabelle et al., 2017b; Burchardt et al., 2017). The paper at hand describes the use of a test suite in order to evaluate 18 German to English and 18 English to German MT systems that partici- pated at the Shared Task of the Sixth Conference on Machine Translation (WMT21)1. The evaluation is performed by an extensive test suite that tests a wide range of linguistically motivated phenom- ena. In addition to our contributions in the previous years, which focused only on German to English, this year we are presenting for the first time results with an extensive test suite with a similar logic for the opposite direction English to German. Our German to English test set contains 5,560 test sen- tences, covering …

Content truncated to 500 words