A Quality Estimation and Quality Evaluation Tool for the Translation Industry

A Quality Estimation and Quality Evaluation Tool for the Translation Industry Elena Murgolo Orbital 14 Milan, Italy emurgolo@orbital14.ai Javad Pourmostafa TSHD, CSAI Department, Tilburg University Tilburg, The Netherlands j.pourmostafa@uvt.nl Dimitar Shterionov TSHD, CSAI Department, Tilburg University, Tilburg, The Netherlands d.shterionov@uvt.nl 1 Introduction With the increase in machine translation (MT) quality over the latest years, it has now become a common practice to integrate MT in the workflow of language service providers (LSPs) and other ac- tors in the translation industry. With MT having a direct impact on the translation workflow, it is im- portant not only to use high-quality MT systems, but also to understand the quality dimension so that the humans involved in the translation work- flow can make informed decisions. The evaluation and monitoring of MT output quality has become one of the essential aspects of language technol- ogy management in LSPs’ workflows. First, a gen- eral practice is to carry out human tests to evaluate MT output quality before deployment. Second, a quality estimate of the translated text, thus after deployment, can inform post-editors or even repre- sent post-editing effort. In the former case, based on the quality assessment of a candidate engine, an informed decision can be made whether the en- gine would be deployed for production or not. In the latter, a quality estimate of the translation out- put can guide the human post-editor or even make rough approximations of the post-editing effort. Quality of an MT engine can be assessed on docu- ment or on sentence level. A tool to jointly provide all these functionalities does not exist yet. While human evaluation is considered the most reliable method of analyzing MT quality, it is time- consuming, expensive, and hardly scalable. Hu- man testing is also difficult to apply for actual projects during a production workflow. While some commercial products that can partly replace human testing already exist, they are usually CAT- © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. dependent and cannot be employed independently from other language technology tools. The overall objective of the project presented in this paper is to develop a machine translation quality assessment (MTQA) tool that simplifies the quality assessment of MT engines, combining quality evaluation and quality estimation on doc- ument and sentence level. To address both use cases, i.e., before general deployment and to esti- mate each translation’s quality, this tool will com- prise two working modes: a machine translation quality evaluation (MTQEv) and a quality estima- tion (MTQE) modes. This 6-month project is a collaboration between Tilburg University and Orbital14, an R&D com- pany owned and 100% financed by Italian LSP Aglatech14, whose funding is making this tool’s development possible. 2 MTQA Tool Overview The MTQA is designed as a standalone tool and an API that can be used by users or invoked by other tools. Behind the user interface lies a distributed architecture which operates in two modes. Inter- mediate and final results …

Content truncated to 500 words