Quality Estimation without Human-labeled Data

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 619–625 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 619 Quality Estimation without Human-labeled Data Yi-Lin Tuan1, Ahmed El-Kishky2, Adithya Renduchintala2, Vishrav Chaudhary2, Francisco Guzm´an2 and Lucia Specia3 1University of California Santa Barbara, 2Facebook AI, 3Imperial College London 1ytuan@cs.ucsb.edu, 2{ahelk,adirendu,vishrav,fguzman}@fb.com 3l.specia@imperial.ac.uk Abstract Quality estimation aims to measure the qual- ity of translated content without access to a reference translation. This is crucial for ma- chine translation systems in real-world scenar- ios where high-quality translation is needed. While many approaches exist for quality es- timation, they are based on supervised ma- chine learning requiring costly human labelled data. As an alternative, we propose a tech- nique that does not rely on examples from human-annotators and instead uses synthetic training data. We train off-the-shelf architec- tures for supervised quality estimation on our synthetic data and show that the resulting mod- els achieve comparable performance to models trained on human-annotated data, both for sen- tence and word-level prediction. 1 Introduction The adoption of Machine Translation (MT) has been increasing in areas ranging from government and finance, to even social media due to the sub- stantial improvements achieved from Neural Ma- chine Translation (NMT). However, even with im- proved performance, translation quality is not con- sistent across language pairs, domains, and sen- tences. This can be detrimental to end-user’s trust and can cause unintended consequences arising from poor translations. Thus, having metrics to as- sess the quality of translated content is crucial to en- sure that only high-quality translations are provided to end-users or downstream tasks. Quality Estima- tion (QE) metrics aim to predict translation quality without access to reference translations (Blatz et al., 2004; Specia et al., 2009, 2013). State-of-the-art QE techniques have leveraged MT systems and language-specific human annota- tions as supervision, including direct assessment and post-editing (Kepler et al., 2019a; Fonseca et al., 2019; Sun et al., 2020). However, these annotations are costly and time-consuming, partic- ularly for word-level QE, where each token needs a label. Some unsupervised approaches take inspiration from statistical MT (Popovi´c, 2012; Moreau and Vogel, 2012; Etchegoyhen et al., 2018) or apply uncertainty quantification (Fomicheva et al., 2020) for QE. However, their performance is inferior to that of supervised models. In related areas such as automatic post-editing, parallel data has been used to create synthetic post-editing data (Negri et al., 2018), however this technique only com- pares machine-translated sentences to references. Our approach augments MT errors with additional errors via masked language model rewriting. We leverage noisy, mined comparable sentences obtained by weakly-supervised techniques (El- Kishky et al., 2020b). These noisy bitexts have been mined from a variety of domains such as Wikipedia (Schwenk et al., 2019a) and large web- crawls (Schwenk et al., 2019b; El-Kishky et al., 2020a; El-Kishky and Guzm´an, 2020) and have been shown to be an invaluable source of train- ing data for NMT models. Using this data is cru- cial to avoid data leakage …

Content truncated to 500 words