Welocalize-ARC/NKUA’s Submission to the WMT 2022 Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 653 - 660 December 7-8, 2022 ©2022 Association for Computational Linguistics Welocalize-ARC/NKUA’s Submission to the WMT 2022 Quality Estimation Shared Task Eirini Zafeiridou1,2, Sokratis Sofianopoulos3 1Welocalize Inc / Frederick, MD, United States 2National and Kapodistrian University of Athens / Athens, Greece 3Institute for Language and Speech Processing - Athena RC / Athens, Greece eirini.zafeiridou@welocalize.com, s_sofian@athenarc.gr Abstract This paper presents our submission to the WMT 2022 quality estimation shared task and more specifically to the quality prediction sentence-level direct assessment (DA) subtask. We build a multilingual system based on the predictor–estimator architecture by using the XLM-RoBERTa transformer for feature extrac- tion and a regression head on top of the final model to estimate the z-standardized DA la- bels. Furthermore, we use pretrained models to extract useful knowledge that reflect various criteria of quality assessment and demonstrate good correlation with human judgements. We optimize the performance of our model by in- corporating this information as additional exter- nal features in the input data and by applying Monte Carlo dropout during both training and inference. 1 Introduction Machine translation quality estimation (MTQE) is the task of automatically estimating the quality of the MT output without using reference translations or any other human input (Blatz et al., 2004; Specia et al., 2009, 2018). MTQE has many use cases and can be applied in various settings (Specia and Shah, 2018). It can be used to estimate the post-editing effort, to rank and compare outputs of different MT systems or to classify the segments that need post-editing. It can also be used to estimate the quality of the final translations as well as to filter out noisy segments from translation memories or training datasets. MTQE techniques usually have multiple granularity levels and can be applied to a word, phrase, sentence or even to an entire doc- ument. Such systems are highly efficient when a vast amount of machine translated segments need to be evaluated in less time, with less effort and lower costs compared to traditional evaluation tech- niques. The WMT 2022 quality estimation shared task includes the following separate tasks: quality pre- diction, explainable QE and critical error detec- tion. Our team participated in the quality predic- tion sentence-level direct assessment (DA) subtask with a multilingual MTQE system. Specifically, we developed a cross-lingual MTQE system following the predictor–estimator architecture (Kim and Lee, 2016; Kim et al., 2017). We used the large-scale pretrained XLM-RoBERTa (XLM-R)1 model (Conneau et al., 2020) for fea- ture extraction, similarly to Chen et al. (2021). We combined the model’s output with additional ex- ternal features that demonstrate good correlation with the target variable. We then used the concate- nated vector as input to our final MTQE regression model. Our regressor is a feed-forward neural net- work with a linear output layer used to estimate the z-standardized DA labels. 2 Quality prediction: sentence-level direct assessment The quality prediction task of the WMT 2022 qual- ity estimation shared task consists of a sentence- …

Content truncated to 500 words