Measuring Uncertainty in Translation Quality Evaluation (TQE)

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1454–1461 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1454 Measuring Uncertainty in Translation Quality Evaluation (TQE) Serge Gladkoff 1 Irina Sorokina 1,2 Lifeng Han 3,5∗ Alexandra Alekseeva 4‡ 1 Logrus Global LLC, 2 Tver State University, 3 The University of Manchester 4 ROKO Labs (‡ deceased) 5 ADAPT Research Centre, Dublin City University {serge.gladkoff, irina.sorokina}@logrusglobal.com * corresponding: lifeng.han@{manchester.ac.uk, adaptcentre.ie} Abstract From both human translators (HT) and machine translation (MT) researchers’ point of view, translation quality evaluation (TQE) is an essential task. Translation service providers (TSPs) have to deliver large volumes of translations which meet customer specifications with harsh constraints of required quality level in tight time-frames and costs. MT researchers strive to make their models better, which also requires reliable quality evaluation. While automatic machine translation evaluation (MTE) metrics and quality estimation (QE) tools are widely available and easy to access, existing automated tools are not good enough, and human assessment from professional translators (HAPs) are often chosen as the golden standard (Han et al., 2021b). Human evaluations, however, are often accused of having low reliability and agreement. Is this caused by subjectivity or statistics is at play? How to avoid the entire text to be checked and be more efficient with TQE from cost and efficiency perspectives, and what is the optimal sample size of the translated text, so as to reliably estimate the translation quality of the entire material? This work carries out such a motivated research to correctly estimate the confidence intervals (Brown et al., 2001) depending on the sample size of translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. The methodology we applied for this work is from Bernoulli Statistical Distribution Modeling (BSDM) and Monte Carlo Sampling Analysis (MCSA). Keywords: Translation Quality Evaluation, Quality Estimation, Post-editing Distance, Confidence Intervals, Monte Carlo Modeling, Bernoulli Statistics 1. Introduction Machine Translation (MT) is one of the pioneering arti- ficial intelligence (AI) tasks dating back from the 1950s (Weaver, 1955). It emphasizes the interaction of lan- guage and machine, and how machine can learn hu- man languages with cognitive knowledge (Han, 2022). Before MT, human translation (HT) of written text and documents has always played an important role in science and literature communication between differ- ent language speakers, breaking the language barriers. From both MT and HT perspectives, translation quality evaluation (TQE), sometimes also referred to as trans- lation quality assessment (TQA) in variation 1, is an important task to reflect how well the source text is translated into the target languages (Han et al., 2021b). On the one hand, for low resource language pair sce- narios, human translators still play the dominant role in translation production. The translated text and doc- uments can contain unavoidable errors due to personal bias, input efforts, or the training level of the transla- tors. On …

Content truncated to 500 words