Test Suite Evaluation: Morphological Challenges and Pronoun Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 458 - 468 December 7-8, 2022 ©2022 Association for Computational Linguistics Test Suite Evaluation: Morphological Challenges and Pronoun Translation Marion Weller-Di Marco and Alexander Fraser Center for Information and Language Processing LMU Munich {dimarco,fraser}@cis.uni-muenchen.de Abstract This paper summarizes the results of our test suite evaluation with a main focus on morphol- ogy for the language pairs English to/from Ger- man. We look at the translation of morpholog- ically complex words (DE–EN), and evaluate whether English noun phrases are translated as compounds vs. phrases into German. Further- more, we investigate the preservation of mor- phological features (gender in EN–DE pronoun translation and number in morpho-syntactically complex structures for DE–EN). Our results in- dicate that systems are able to interpret linguis- tic structures to obtain relevant information, but also that translation becomes more challenging with increasing complexity, as seen, for exam- ple, when translating words with negation or non-concatenative properties, and for the more complex cases of the pronoun translation task. 1 Introduction Evaluating MT output is challenging. Document- levels metrics give a rather coarse-grained estima- tion of the overall translation quality, but cannot determine how well a system operates for particular challenges. Translations do not have a determin- istic solution, but there are always several possi- bilities for a valid translation, making a focused evaluation of particular phenomena difficult. The annual WMT Shared Task provides the pos- sibility to submit custom test suites to be translated in addition to the regular test sets, which allows the investigation of the translation performance of state-of-the-art systems when presented with par- ticular translation tasks. In this test suite, we focus on morphological challenges for English to/from German translation: For German–English transla- tion, we look at the translation of morphologically complex words, in addition to a small set of sen- tences where a subtle difference (singular vs. plu- ral) needs to be detected. For English–German, we study how complex noun phrases are translated – as compounds or rather as multi-word phrases. Fur- thermore, we add a pronoun translation task and evaluate the translation of the English pronoun it into its German equivalents er/sie/es, depending on the gender of the noun it refers to. The test suite does not aim at measuring a sys- tem’s general translation performance – this is al- ready assessed by means of a manual evaluation and various other metrics in the main shared task – but rather at evaluating the translational behaviour for carefully selected words or phrases. As the sen- tences in the test suite are not parallel, we opt for a semi-automatic approach where translation options for the words in question are manually collected and then matched with the translation output. Thus, only the translation of the relevant word is consid- ered, whereas the rest of the sentence is ignored. 2 Data Creation and Evaluation In the following, we outline the process of compos- ing and evaluating the test suite. Selection of words The sets for the analysis of …

Content truncated to 500 words