Findings of the WMT 2022 Shared Task on Chat Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 724 - 743 December 7-8, 2022 ©2022 Association for Computational Linguistics Findings of the WMT 2022 Shared Task on Chat Translation Ana C. Farinha1⇤ M. Amin Farajian1⇤ Marianna Buchicchio1 Patrick Fernandes4,5,6 José G. C. de Souza1 Helena Moniz1,2,3 André F. T. Martins1,4,5 1Unbabel, Lisbon, Portugal 2INESC-ID, Lisbon, Portugal 3Faculdade de Letras, University of Lisbon, Portugal 4Instituto de Telecomunicações, Lisbon, Portugal 5Instituto Superior Técnico, University of Lisbon, Portugal 6Carnegie Mellon University, Pittsburgh, USA Abstract This paper reports the findings of the sec- ond edition of the Chat Translation Shared Task. Similarly to the previous WMT 2020 edi- tion, the task consisted of translating bilingual customer support conversational text. How- ever, unlike the previous edition, in which the bilingual data was created from a syn- thetic monolingual English corpus, this year we used a portion of the newly released Un- babel’s MAIA corpus, which contains genuine bilingual conversations between agents and cus- tomers. We also expanded the language pairs to English$German (en$de), English$French (en$fr), and English$Brazilian Portuguese (en$pt-br). Given that the main goal of the shared task is to translate bilingual conversations, participants were encouraged to train and test their mod- els specifically for this environment. In total, we received 18 submissions from 4 different teams. All teams participated in both directions of en$de. One of the teams also participated in en$fr and en$pt-br. We evaluated the submis- sions with automatic metrics as well as human judgments via Multidimensional Quality Met- rics (MQM) on both directions. The official ranking of the systems is based on the overall MQM scores of the participating systems on both directions, i.e. agent and customer. 1 Introduction With the significant translation quality improve- ments brought by newer machine translation (MT) approaches in the last years, we can start using MT to translate non-conventional content types such as bilingual and multilingual conversations. These new applications pose new challenges to MT sys- tems and require new solutions to deal with them. ⇤These authors contributed equally. In this shared task, we focus on the automatic trans- lation of conversational text, in particular customer support chats, an important and challenging content due to their particular characteristics (Gonçalves et al., 2022; Wang et al., 2021; Farajian et al., 2020): In contrast to content types such as news articles and software manuals, among others, in which the text is carefully authored and well formatted, chat conversations are less planned, more informal, and often present ungrammatical linguistic structures. Furthermore, such conversations are usually on- the-fly production of text with very fuzzy fron- tiers with speech and mimicking speech production. Due to time requirements, since in dialogues turn- exchange need to be dynamic, the conversations may also have typos, abbreviations and ellipses. The conversations are also characterized by stress- ful moments, which in turn is represented by the capitalization of the entire word or turn, emoticons or emojis, and multiple punctuation marks. Furthermore, Gonçalves et al. describe several factors that often lead to poor quality …

Content truncated to 500 words