Linguistically Motivated Evaluation of Machine Translation Metrics Based on a Challenge Set

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 514 - 529 December 7-8, 2022 ©2022 Association for Computational Linguistics Linguistically Motivated Evaluation of Machine Translation Metrics based on a Challenge Set Eleftherios Avramidis and Vivien Macketanz German Research Center for Artificial Intelligence (DFKI), Speech and Language Technology, Berlin, Germany firstname.lastname@dfki.de Abstract We employ a linguistically motivated chal- lenge set in order to evaluate the state-of-the- art machine translation metrics submitted to the Metrics Shared Task of the 7th Conference for Machine Translation. The challenge set includes about 20,000 items extracted from 145 MT systems for two language directions (German,English), covering more than 100 linguistically-motivated phenomena organized in 14 categories. The best performing met- rics are YiSi-1, BERTScore and COMET-22 for German-English, and UniTE, UniTE-ref, MetricX-XL-DA19 and MetricX-XXL-DA19 for English-German. Metrics in both directions are performing worst when it comes to named- entities & terminology and particularly mea- suring units. Particularly in German-English they are weak at detecting issues at punctuation, polar questions, relative clauses, dates and id- ioms. In English-German, they perform worst at present progressive of transitive verbs, fu- ture II progressive of intransitive verbs, simple present perfect of ditransitive verbs and focus particles. 1 Introduction Automatic evaluation metrics have been valuable tools for Machine Translation (MT), allowing quick evaluation and suggesting directions for further de- velopment. Many metrics have been suggested throughout the years, which in turn sets the require- ment for their evaluation. Whereas MT metrics so far have been evaluated based on the agreement of their scores with human judgments on test sets drawn from broad text, little research has taken place on investigating whether the performance of the metrics generalizes enough when evaluating particular cases. A more target way of evaluating metrics is using challenge sets. These are targeted test sets, which have been de- vised in such a way, so that they benchmark the ability of metrics to score particular translation phe- nomena. In this paper we present empirical results on the performance of MT metrics, using an exten- sive challenge set, which includes thousands of test items aiming to test the performance over more than one hundred linguistically-motivated phenom- ena in two language directions. It is based on thou- sands of manually created test items, their trans- lation outputs from dozens of MT systems and semi-automatically evaluated with the supervision of linguists. Through this analysis we attempt to reveal strengths and weaknesses of several state-of- the-art MT metrics considering their background methods with regards to linguistic aspects. The rest of the paper is structured as follows. In Section 2 related work is briefly described. In Section 3 we describe the construction of the chal- lenge set and the evaluation protocol. The empiri- cal results are outlined in Section 4, followed by a conclusion is Section 5. 2 Related work The need for a thorough evaluation of Natural Lan- guage Processing (NLP) tools has lately received increased interest in the research community, indi- cated by a big amount of publications, among them …

Content truncated to 500 words