Semi-supervised Learning for Quality Estimation of Machine Translation

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 72–83 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 72 Semi-supervised Learning for Quality Estimation of Machine Translation Tarun Bhatia tarunbhatia.ind@gmail.com Technische Universität Berlin, Berlin, Germany and SAP SE Martin Krämer martin.kraemer@sap.com Eduardo Vellasques eduardo.vellasques@sap.com SAP SE Eleftherios Avramidis eleftherios.avramidis@dfki.de German Research Center for Artificial Intelligence (DFKI), Berlin, Germany Abstract We investigate whether using semi-supervised learning (SSL) methods can be beneficial for the task of word-level Quality Estimation of Machine Translation in low-resource conditions. We show that the Mean Teacher network can provide equal or significantly better MCC scores (up to +12%) than supervised methods when a limited amount of labeled data is available. Additionally, following previous work on SSL, we investigate Pseudo-Labeling in combination with SSL, which nevertheless does not provide consistent improvements. 1 Introduction Through the recent development of Machine Translation (MT), Quality Estimation (QE) has come to serve the need to predict the quality of translation provided by MT systems when no reference translations are available. QE has been mostly treated as a supervised learning problem, where supervised models can be trained on the source and translated text along with their respective quality labels. For example, QE at the word level includes the source and translated sentences as the data and their label sequence includes an OK or BAD label for each translated word in the sentence, which can determine if the word is correctly translated or not and potential errors in the translations can be flagged. In order to train supervised models for such problems, a large amount of labeled data is needed. However, such data is expensive to create as it involves human annotators to post-edit or generate labels for the given translations. Whereas the unavailability of labeled data is a problem, there is an abundance of unlabeled data for such a task, i.e. source sentences and the corresponding translations generated by MT systems. Semi-supervised learning (SSL) methods could be utilized to train QE models with few labeled data available along with unlabeled data that can be generated in abundance. While the prominent SSL approach of Mean Teacher has shown good performance in computer vision (Tarvainen and Valpola, 2017), there has been little experimentation in NLP. Until now, no research has followed SSL to fine-tune pre-trained language models (LMs) for the task of QE of MT. This work focuses on implementing the aforementioned SSL strategies for word-level QE and tries to answer the following questions: 73 Figure 1: Example of German translation by a MT system and its human post-edited version with labels of gaps and target words for the example translated text. Tags for Gap tokens are highlighted in yellow and tags for words in sequence are highlighted in blue (Specia et al., 2021) 1. Can the SSL method of Mean Teacher perform equal or better than supervised methods on low-resource conditions? 2. Is it possible to …

Content truncated to 500 words