Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5865–5875 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5865 Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications Shuo Sun,1 Ahmed El-Kishky,2 Vishrav Chaudhary,3 James Cross, 3 Francisco Guzmán,3 Lucia Specia4 1Johns Hopkins University, 2Twitter Cortex, 3Facebook AI, 4Imperial College London 1ssun32@jhu.edu, 2aelkishky@twitter.com 3{vishrav,jcross,fguzman}@fb.com, 4l.specia@imperial.ac.uk Abstract Sentence-level Quality Estimation (QE) of ma- chine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson cor- relation with human labels. Recent QE mod- els have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized lan- guage models that are computationally expen- sive and thus infeasible for many real-world applications. In this work, we evaluate sev- eral model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this re- gression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we ar- gue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual perfor- mance in real-world applications. 1 Introduction Quality Estimation (QE) (Specia et al., 2020) is the task of predicting the quality of an automatically translated sentence at test time, without the need to rely on reference translations. Formally, given a source sentence, s and a translated sentence, t, the goal of QE is to learn a regression model, m such that m(s, t) returns a score that represents the quality of the translated sentence. There are many important applications of qual- ity estimation, for example, translation companies use QE systems to identify mistranslated sentences, which would reduce post-editing costs and efforts, while online platforms use QE systems as filters to hide poorly translated sentences from end-users. Additionally, with the proliferation of mined par- allel data obtained from web-crawls as a source of NMT training data (El-Kishky et al., 2020b,a), QE has become an important tool in performing quality control on translations from models trained on noisy training data. The performance of a QE system is usually mea- sured by the correlation between predicted QE and human-annotated QE scores. However, the pre- dictions of QE models are primarily used to make binary decisions (Zhou et al., 2020): only transla- tions above a certain QE threshold would be given to a human for post-edition in a translation com- pany, or shown to the user in an online platform. Therefore, Pearson correlation might not be the best metric to evaluate the actual performance of the QE models in real-world use cases. In recent iterations of the QE shared task at the Conference on Machine Translation (WMT) (Fonseca et al., 2019; Specia et al., 2020), the top- performing QE systems have been built …

Content truncated to 500 words