A Taxonomy and Study of Critical Errors in Machine Translation

A Taxonomy and Study of Critical Errors in Machine Translation Khetam Al Sharou1 and Lucia Specia1, 2 1Language and Multimodal AI Lab, Imperial College London, UK 2Computer Science Department, University of Sheffield, UK {k.al-sharou, l.specia}@imperial.ac.uk Abstract Not all machine mistranslations are of equal scale of severity. For example, mis- translating a date or time in an appoint- ment, mistranslating a number or currency in a contract, or hallucinating profanity may lead to catastrophic consequences for the users. The severity of the errors is an important but overlooked aspect of ma- chine translation (MT) quality evaluation. In this paper, we present the results of our effort to bring awareness to the problem of critical translation errors. We study, val- idate and extend an initial taxonomy of critical errors with the view of providing guidance for critical error analysis, anno- tation and mitigation. We test the extended taxonomy for three language pairs to ex- amine to what extent it generalises across languages. We provide an account of fac- tors that affect annotation tasks along with recommendations on how to improve an- notation practice in future work. We also study patterns in the source text that can lead to critical errors. Detecting such lin- guistic patterns could be used to improve the performance of MT systems, especially for user-generated content. 1 Introduction Machine Translation (MT) has now become ubiq- uitous in many online platforms (e.g. social net- works) and generally used without any human post-editing due to cost, timeliness, and accessi- bility. The rapid development and adoption of MT © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. has advanced efforts to improve and standardise MT evaluation, and increased discussion on how we should evaluate MT (Dorr et al., 2011; García, 2014; Ulitkin et al., 2021). This need escalated with the use of MT to translate user-generated con- tent (UGC), e.g. in social media platforms. Un- like formal text, UGC often has colloquial lan- guage, including profanities, spelling errors, emo- jis, hashtags and abbreviations, and is grammati- cally ill-formed, which makes it hard for MT, of- ten resulting in incorrect translations (Al Sharou et al., 2021). Some of these incorrect translations can contain critical errors. In this work, we refer to critical errors as instances of translations where the meaning in the target text deviates drastically from the source text where such translations can be misleading and may carry health, safety, legal, reputation, religious or financial implications. The volume of content shared by users means that the MT-translated content cannot be manually post-edited. Therefore, users have to rely on MT as is and usually do not have the linguistic skills to identify the errors. As a consequence, users may be negatively affected if they misunderstand the intention or sentiment of the source text or could take inappropriate action if they act on critically corrupted translations. There are many instances where innocuous statements on social media have been translated by the machine to …

Content truncated to 500 words