A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 286–297 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 286 A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation Miguel Menezes INESC-ID, Unbabel, University of Lisbon, Lisbon, Portugallmenezes@campus.ul.pt Amin Farajian amin@unbabel.com Unbabel, Lisbon, Portugal Helena Moniz helena@unbabel.com INESC-ID, Unbabel, University of Lisbon, Lisbon, Portugal Jo˜ao Grac¸a joao@unbabel.com Unbabel, Lisbon, Portugal Abstract To measure context-aware machine translation (MT) systems quality, existing solutions have recommended human annotators to consider the full context of a document. In our work, we revised a well known Machine Translation quality assessment framework, Multidimensional Quality Metrics (MQM), (Lommel et al., 2014) by introducing a set of nine annotation categories that allows to map MT errors to source document contextual phenomenon, for simplicity sake we named such phenomena as contextual triggers. Our analysis shows that the adapted categories set enhanced MQM’s potential for MT error identification, being able to cover up to 61% more errors, when compared to traditional non-context core MQM’s application. Subsequently, we analysed the severity of these MT “contextual errors”, showing that the majority fall under the critical and major levels, further indicating the impact of such errors. Finally, we measured the ability of existing evaluation metrics in detecting the proposed MT “contextual errors”. The results have shown that current state-of-the-art metrics fall short in detecting MT errors that are caused by contextual triggers on the source document side. With the work developed, we hope to understand how impactful context is for enhancing quality within a MT workflow and draw attention to future integration of the proposed contextual annotation framework into current MQM’s core typology. Keywords Context-aware error typologies; Machine Translation; Customer Support; Test-Suites; Trans- lation Quality Workflows and Automation; Automatic metrics. 1 Introduction In past decades, the staggering growth in demand for shared knowledge has led to an increase in translation requests, exceeding human translators’ work capacity. In order to accommodate to such request, many enterprises are now integrating MT systems to their workflow that allegedly provide human-like translations in record time. However, despite often claims of 287 human-parity (Xiong et al., 2017), there are plenty of work in the field (Wan et al., 2022; Singh and Singh, 2022) that dispel such allegations, even showing that, under certain circumstances, state-of-the art conventional approaches under-perform and are unable to deal with language nuances, translating words instead of “meanings”. Aware of Neural Machine Translation (NMT) limitations, in the last few years, new approaches have been devised to leverage document context for finer-grained MT outputs. Despite sharing similar believes, we suspect that researchers have only now begun to scratch the surface on such complex subject matter, especially when it is not yet clear that context-aware MT systems are indeed able to account for context within a document (Yin et al., 2021). Yet, there is scarce research into document-level MT quality assessment (QA) metrics for more reliable evaluations (Castilho …

Content truncated to 500 words