Evaluating and Improving the Coreference Capabilities of Machine Translation Models

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 980–992 May 2-6, 2023 ©2023 Association for Computational Linguistics Evaluating and Improving the Coreference Capabilities of Machine Translation Models Asaf Yehudai1 Arie Cattan2 Omri Abend1 Gabriel Stanovsky1 1School of Computer Science, The Hebrew University of Jerusalem 2Computer Science Department, Bar Ilan University {asaf.yehudai,omri.abend,gabriel.stanovsky}@mail.huji.ac.il arie.cattan@gmail.com Abstract Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implic- itly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and eval- uates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the- art coreference resolvers on three challenging benchmarks. Our results show that the monolin- gual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, show- ing improvement over strong baselines.1 1 Introduction Machine translation (MT) may require coreference resolution to translate cases where the source and target language differ in their grammatical proper- ties. For example, consider translating “The trophy didn’t fit in the suitcase because it was too small” from English to French: “Le trophée ne rentrait pas dans la valise car elle était trop petite” (Sak- aguchi et al., 2020). In French, suitcase (“valise”) is grammatically feminine, and trophy (“trophée”) is masculine, while the source-side English does not encode grammatical noun gender. This re- quires an MT model to infer that “it” refers to the suitcase (and not the trophy) to correctly pro- duce the feminine inflection for the phrase “it was too small” (“elle était trop petite”), whereas an incorrect coreference resolution may produce the masculine inflection (“il était trop petit”) corre- sponding to the trophy. 1https://github.com/AsafYehudai/MT-coref Figure 1: MT models can be compared to source-side coreference resolvers. An example translation from English (turquoise) to French (yellow). Our method first identifies the grammatical gender of the mentions in the target language marked in purple (female) and blue (male), followed by inferring the source side clusters (orange), through gender agreement. Such texts evade lexical one-to-one translation, and instead demand source-side coreference resolu- tion as a prerequisite for a correct translation. The prominent end-to-end approach to MT assumes that translation models implicitly learn source-side coreference resolution by observing aligned source- target pairs, without intermediate coreference su- pervision. While the importance of addressing such semantic phenomenon has been stated in various works (Le Nagard and Koehn, 2010; Stojanovski and Fraser, 2018), it was also observed that the ubiquitous BLEU metric (Papineni et al., 2002) does not adequately quantify it (Hardmeier and Federico, 2010; Freitag et al., 2022). 980 This work addresses the following research ques- tion: How well does MT learn coreference when …

Content truncated to 500 words