On nature and causes of observed MT errors

On nature and causes of observed MT errors Maja Popovi´c maja.popovic@adaptcentre.ie ADAPT Centre, School of Computing, Dublin City University, Ireland Abstract This work describes analysis of nature and causes of MT errors observed by different evalua- tors under guidance of different quality criteria: adequacy, comprehension, and a not specified generic mixture of adequacy and fluency. We report results for three language pairs, two do- mains and eleven MT systems. Our findings indicate that, despite the fact that some of the identified phenomena depend on domain and/or language, the following set of phenomena can be considered as generally challenging for modern MT systems: rephrasing groups of words, translation of ambiguous source words, translating noun phrases, and mistranslations. Further- more, we show that the quality criterion also has impact on error perception. Our findings indicate that comprehension and adequacy can be assessed simultaneously by different evalu- ators, so that comprehension, as an important quality criterion, can be included more often in human evaluations. 1 Introduction and related work Machine translation (MT), like many other natural language generation tasks, is difficult to evaluate because there is no single correct output for a given input: for each source text, there is a large set of possible correct translations. Therefore, while costly both in time and resources, human evaluation is required to provide a reliable feedback for measuring MT quality and progress, as well as to serve as a gold standard for development of automatic evaluation metrics. While better and better automatic metrics are constantly emerging (Mathur et al., 2020; Ma et al., 2019), many of them being based on semantic word representations (embeddings), all of them represent only an approximate substitution for human assessment of translation quality. Various methods have been proposed and used for the human evaluation of MT output from the beginning of MT until now (ALPAC, 1966; White et al., 1994; Koehn and Monz, 2006; Vilar et al., 2007; Graham et al., 2013; Forcada et al., 2018; Barrault et al., 2020; Kreutzer et al., 2020; Popovi´c, 2020a), and all of them are essentially based on some of the following three quality criteria: adequacy (how much meaning is preserved), comprehensibility (how comprehensible/readable the translation is) and fluency (grammar of the target language). The evaluators are usually asked to assign an overall quality score for the given MT out- put (ALPAC, 1966; White et al., 1994; Koehn and Monz, 2006; Roturier and Bensadoun, 2011; Graham et al., 2013; Barrault et al., 2020) or to rank two or more competing outputs from best to worst (Vilar et al., 2007; Callison-Burch et al., 2008; Bojar et al., 2015). For assessing compre- hension, question answering (Scarton and Specia, 2016) and filling gaps (Forcada et al., 2018) were explored, too. Recently, evaluators have been asked to highlight the observed translation errors (Kreutzer et al., 2020; Popovi´c, 2020a). In order to get more details about the actual errors, error classification according to a prede- fined error scheme is often performed. The mostly applied schemes have been the one …

Content truncated to 500 words