QUARTZ: Quality-Aware Machine Translation

QUARTZ: Quality-Aware Machine Translation José G. C. de Souza1 Ricardo Rei1,2,4 Ana C Farinha1 Helena Moniz1,4,5 André F. T. Martins1,2,3 1Unbabel 2Instituto Superior Técnico 3Instituto de Telecomunicações 4INESC-ID 5Faculdade de Letras da Universidade de Lisboa Lisbon, Portugal jose.souza, ricardo.rei, catarina.farinha, helena.moniz, andre.martins@unbabel.com Abstract This paper presents QUARTZ, QUality- AwaRe machine Translation, a project led by Unbabel and funded by the ELISE Open Call1 which aims at developing ma- chine translation systems that are more robust and produce fewer critical errors. With QUARTZ we want to enable ma- chine translation for user-generated con- versational content types that do not tol- erate critical errors in automatic transla- tions. The project runs from January to July 2022. 1 Introduction Despite the progress in the fluency of machine translation (MT) systems, critical translation errors are still frequent, including deviations in meaning through toxic or offensive content, hallucinations, mistranslation of entities with health, safety, or fi- nancial implications, or deviation in sentiment po- larity or negation. These errors occur more often when the source sentence is out of domain or con- tains typos, abbreviations, or capitalized text, all common with user-generated content. This lack of robustness prevents the use of MT systems in prac- tical applications where the above errors cannot be tolerated. QUARTZ aims to build reliable, quality-aware MT systems for user-generated conversational data. The project will address the limitations above by: (a) developing quality metrics capable of de- tecting critical errors and hallucinations; (b) en- dowing MT systems with a confidence (quality) © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1Funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 951847 score, and fine-tuning pre-trained MT models to the domains in which they will be used through quality-driven objectives. This will be done by leveraging post-edited data and quality annotations produced by the Unba- bel community and building upon the state-of-the- art, open-source quality estimation technology al- ready existing at Unbabel: OPENKIWI (Kepler et al., 2019) and COMET (Rei et al., 2020). From a product perspective, focus will be given to con- versational, user-generated data in a multilingual customer service scenario (email or chat involving a customer and an agent), in which Unbabel has renowned expertise and existing technology vali- dated by existing customers. The solution aims to eliminate language barriers in the highly multilin- gual European market. 2 MT and Translation Quality The current state of the art in MT is based on auto- regressive sequence-to-sequence models trained with maximum likelihood and teacher forcing. This objective encourages the model to assign high probability to reference translations, but does not account for the severity of translation mistakes of the hypotheses generated. This leads to exposure bias, vulnerability to adversarial attacks, and no control for hallucinations, harmful content, and bi- ases (Wang and Sennrich, 2020), hampering the responsible use of NMT for user-generated con- versational content. Project Overview Qualitative evaluation car- ried out by translators (post-editors and …

Content truncated to 500 words