Ensemble Fine-tuned mBERT for Translation Quality Estimation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 897–903 November 10–11, 2021. ©2021 Association for Computational Linguistics 897 Ensemble Fine-tuned mBERT for Translation Quality Estimation Shaika Chowdhury ∗ University of Illinois at Chicago, US schowd21@uic.edu Naouel Baili IQVIA, US naouel.baili@iqvia.com Brian Vannah IQVIA, US brian.vannah@iqvia.com Abstract Quality Estimation (QE) is an important com- ponent of the machine translation workflow as it assesses the quality of the translated out- put without consulting reference translations. In this paper, we discuss our submission to the WMT 2021 QE Shared Task. We partici- pate in Task 2 sentence-level sub-task that chal- lenge participants to predict the HTER score for sentence-level post-editing effort. Our proposed system is an ensemble of multilin- gual BERT (mBERT)-based regression mod- els, which are generated by fine-tuning on dif- ferent input settings. It demonstrates compara- ble performance with respect to the Pearson’s correlation and beats the baseline system in MAE/ RMSE for several language pairs. In ad- dition, we adapt our system for the zero-shot setting by exploiting target language-relevant language pairs and pseudo-reference transla- tions. 1 Introduction Progress in machine translation (MT) has accel- erated due to the introduction of deep learning based approaches, dubbed as neural machine trans- lation (NMT) (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014). Several metrics (e.g., BLEU (Papineni et al., 2002), METEOR (Agar- wal and Lavie, 2008)) are used to automatically evaluate the quality of the translations outputted by the NMT systems. However, these evaluation met- rics require comparing the NMT outputs against human-prepared reference translations, which can- not be readily obtained. To tackle this predicament, recently quality estimation (QE) (Blatz et al., 2004; Specia et al., 2018) has emerged as an alternative evaluation approach for NMT systems. QE obvi- ates the need for human judgements and hence can be efficiently integrated into the dynamic transla- tion pipeline in the industry setting. ∗work done during internship at IQVIA QE is performed at different granularity (e.g., word, sentence, document) (Kepler et al., 2019a); in this work we focus on the sentence-level post- editing effort task, which predicts the quality of the translated sentence as a whole in terms of the number of edit operations that need to be made to yield a post-edited translation, termed as HTER (Snover et al., 2006). Sentence-level QE using neural approaches is generally treated as a supervised regression prob- lem involving mainly two steps. In the first step, an encoder is used to learn vector representation/s of the source and translation sentences. While in the second step, the learned representations are passed through a sigmoid output layer to estimate the HTER score. These two steps can be per- formed either with a single model in an end-to- end fashion (e.g., Bi-RNN (Ive et al., 2018)), or using two separate models (e.g., POSTECH (Kim et al., 2017)). The different QE systems vary in their choice of the encoder, which range from RNN- based to Transformer-based models. In this work, we leverage the fine-tuning capa- …

Content truncated to 500 words