QEMind: Alibaba’s Submission to the WMT21 Quality Estimation Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 948–954 November 10–11, 2021. ©2021 Association for Computational Linguistics 948 QEMind: Alibaba’s Submission to the WMT21 Quality Estimation Shared Task Jiayi Wang∗, Ke Wang*, Boxing Chen, Yu Zhao, Weihua Luo, Yuqi Zhang† Alibaba Group, Hangzhou, China {joanne.wjy,moyu.wk,boxing.cbx}@alibaba-inc.com, kongyu@taobao.com, {weihua.luowh,chenwei.zyq}@alibaba-inc.com Abstract Quality Estimation, as a crucial step of quality control for machine translation, has been ex- plored for years. The goal is to investigate au- tomatic methods for estimating the quality of machine translation results without reference translations. In this year’s WMT QE shared task, we utilize the large-scale XLM-Roberta pre-trained model and additionally propose several useful features to evaluate the uncer- tainty of the translations to build our QE sys- tem, named QEMind. The system has been ap- plied to the sentence-level scoring task of Di- rect Assessment and the binary score predic- tion task of Critical Error Detection. In this pa- per, we present our submissions to the WMT 2021 QE shared task and an extensive set of experimental results have shown us that our multilingual systems outperform the best sys- tem in the Direct Assessment QE task of WMT 2020. 1 Introduction Quality estimation (QE) aims to predict the qual- ity of a machine translation (MT) system’s output without any access to ground-truth translation ref- erences or human intervention (Blatz et al., 2004; Specia et al., 2009, 2018). Automatic methods for QE are highly appreciated in MT applications when we expect to efficiently obtain the quality indica- tions for a larget amount of machine translation out- puts in a short time, or even at run-time. This paper describes Alibaba’s submissions to the WMT 2021 Quality Estimation Shared Task. We developed a novel QE system, called QEMind, that have been applied to two tasks this year, the sentence-level direct assessment (DA) and binary score prediction of Critical Error Detection (CED). Common approaches in the previous years heav- ily focus on human-crafted rule-based feature engi- neering mode such as QuEst++ (Specia et al., 2015). ∗ indicates equal contribution. † indicates corresponding author. XLM-Roberta [CLS] Output Feed Forward Layer Output Projection Scoring for Direct Assessment or Binary Classification for Critical Error Detection Source Text Machine Translation + Uncertainty Features Figure 1: Structure of the uncertainty quantification feature-enhanced model. The features extracted are usually fed into tradi- tional machine learning algorithms such as a sup- port vector regression for the sentence-level scor- ing or a sequence-labeling model with conditional random fields for the word-level labeling respec- tively. With the development of neural networks applied in machine translation and other NLP tasks, a neural predictor-estimator framework for QE was proposed and achieved better results in WMT 2017 and WMT 2018 QE shared tasks (Fan et al., 2019; Kim et al., 2017). This framework extensively re- quires a pre-training procedure with a large amount of parallel corpora in the predictor mode and stacks a downstream estimator mode with additional lay- ers for a supervised regression or classification task. Since 2019, state-of-the-art (SOTA) QE systems …

Content truncated to 500 words