TransQuest: Translation Quality Estimation with Cross-lingual Transformers

Proceedings of the 28th International Conference on Computational Linguistics, pages 5070–5081 Barcelona, Spain (Online), December 8-13, 2020 5070 TransQuest: Translation Quality Estimation with Cross-lingual Transformers Tharindu Ranasinghe♦, Constantin Or˘asan♥ and Ruslan Mitkov♦ ♦Research Group in Computational Linguistics, University of Wolverhampton, UK ♥Centre for Translation Studies, University of Surrey, UK {t.d.ranasinghehettiarachchige, r.mitkov}@wlv.ac.uk c.orasan@surrey.ac.uk Abstract Recent years have seen big advances in the field of sentence-level quality estimation (QE), largely as a result of using neural-based architectures. However, the majority of these methods work only on the language pair they are trained on and need retraining for new language pairs. This process can prove difficult from a technical point of view and is usually computationally expensive. In this paper we propose a simple QE framework based on cross-lingual transformers, and we use it to implement and evaluate two different neural architectures. Our evaluation shows that the proposed methods achieve state-of-the-art results outperforming current open-source quality estimation frameworks when trained on datasets from WMT. In addition, the framework proves very useful in transfer learning settings, especially when dealing with low-resourced languages, allowing us to obtain very competitive results. 1 Introduction The goal of quality estimation (QE) is to evaluate the quality of a translation without having access to a reference translation (Specia et al., 2018). High-accuracy QE that can be easily deployed for a number of language pairs is the missing piece in many commercial translation workflows as they have numerous potential uses. They can be employed to select the best translation when several translation engines are available or can inform the end user about the reliability of automatically translated content. In addition, QE systems can be used to decide whether a translation can be published as it is in a given context, or whether it requires human post-editing before publishing or even translation from scratch by a human (Kepler et al., 2019). The estimation of translation quality can be done at different levels: document level, sentence level and word/phrase level (Ive et al., 2018). In this research we focus on sentence-level quality estimation. As we discuss in Section 2, at present neural-based QE methods constitute the state of the art in quality estimation. However, these approaches are based on complex neural networks and require resource- intensive training. This resource-intensive nature of these deep-learning-based frameworks makes it expensive to have QE systems that work for several languages at the same time. Furthermore, these architectures require a large number of annotated instances for training, making the quality estimation task very difficult for low-resource language pairs. In this paper we propose TransQuest, a framework for sentence-level machine translation quality estimation which solves the aforementioned problems, whilst obtaining competitive results. The motivation behind this research is to propose a simple architecture which can be easily trained with different types of inputs (i.e. different language pairs or language from different domains) and can be used for transfer learning in settings where there is not enough training data. We show that TransQuest outperforms current open-source quality estimation frameworks …

Content truncated to 500 words