Using Linguistic Typology to Enrich Multilingual Lexicons: the Case of Lexical Gaps in Kinship

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 13–21 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 13 HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation Serge Gladkoff 1, Lifeng Han 2,3 1 Logrus Global LLC, Pennsylvania, United States 2 The University of Manchester, United Kingdom & 3 ADAPT Centre, Dublin City University, Ireland serge.gladkoff@logrusglobal.com & lifeng.han@{manchester.ac.uk, adaptcentre.ie} Abstract Traditional automatic evaluation metrics for machine translation have been widely criticized by linguists due to their low accuracy, lack of transparency, focus on language mechanics rather than semantics, and low agreement with human quality evaluation. Human evaluations in the form of MQM-like scorecards have always been carried out in real industry setting by both clients and translation service providers (TSPs). However, traditional human translation quality evaluations are costly to perform and go into great linguistic detail, raise issues as to inter-rater reliability (IRR) and are not designed to measure quality of worse than premium quality translations. In this work, we introduce HOPE, a task-oriented and human-centric evaluation framework for machine translation output based on professional post-editing annotations. It contains only a limited number of commonly occurring error types, and uses a scoring model with geometric progression of error penalty points (EPPs) reflecting error severity level to each translation unit. The initial experimental work carried out on English-Russian language pair MT outputs on marketing content type of text from highly technical domain reveals that our evaluation framework is quite effective in reflecting the MT output quality regarding both overall system-level performance and segment-level transparency, and it increases the IRR for error type interpretation. The approach has several key advantages, such as ability to measure and compare less than perfect MT output from different systems, ability to indicate human perception of quality, immediate estimation of the labor effort required to bring MT output to premium quality, low-cost and faster application, as well as higher IRR. Our experimental data is available at https://github.com/lHan87/HOPE. Keywords: Machine Translation Evaluation, Professional Post-Editing, Human Evaluation, Error Classifications 1. Introduction Recent studies show that human evaluation of machine translation (MT) output quality is the gold standard of translation quality evaluation, since no automated met- rics can achieve equally significant results (Freitag et al., 2021; Han et al., 2020b; Han et al., 2021a; Han, 2022). However, existing advanced methods of human quality evaluations, although well developed, such as Multidimensional Quality Metrics (MQM) (Lommel et al., 2014), have the following drawbacks: 1) they are very time- and effort-consuming; 2) they are making it difficult to address the specific needs of MT post- editing, where target quality in many cases is expected to be of what TAUS (the Translation Automation User Society) has coined as “good enough” quality, which is substandard by the premium quality metrics, because they are designed to evaluate a near-premium quality material; 3) they track too many linguistic details that are unnecessary for MT output quality evaluation; 4) do not track …

Content truncated to 500 words