An Exploratory Study on Multilingual Quality Estimation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 366–377 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 366 An Exploratory Study on Multilingual Quality Estimation Shuo Sun,1* Marina Fomicheva,2* Fr´ed´eric Blain,2 Vishrav Chaudhary,3 Ahmed El-Kishky,3 Adithya Renduchintala,3 Francisco Guzm´an,3 Lucia Specia2,4 1Johns Hopkins University, 2University of Sheffield, 3Facebook AI 4Imperial College London 1ssun32@jhu.edu 2{m.fomicheva,f.blain,l.specia}@sheffield.ac.uk 3{vishrav,ahelk,adirendu,fguzman}@fb.com Abstract Predicting the quality of machine transla- tion has traditionally been addressed with language-specific models, under the assump- tion that the quality label distribution or lin- guistic features exhibit traits that are not shared across languages. An obvious disadvan- tage of this approach is the need for labelled data for each given language pair. We chal- lenge this assumption by exploring different approaches to multilingual Quality Estimation (QE), including using scores from translation models. We show that these outperform single- language models, particularly in less balanced quality label distributions and low-resource settings. In the extreme case of zero-shot QE, we show that it is possible to accurately predict quality for any given new language from mod- els trained on other languages. Our findings indicate that state-of-the-art neural QE models based on powerful pre-trained representations generalise well across languages, making them more applicable in real-world settings. 1 Introduction Quality Estimation (QE) (Blatz et al., 2004a; Spe- cia et al., 2009) is the task of predicting the quality of an automatically generated translation at test time, when no reference translation is available for comparison. Instead of reference translations, QE turns to explicit quality indicators that are either provided by the Machine Translation (MT) system itself (the so-called glass-box features) or extracted from both the source and the target texts (the so- called black-box features) (Specia et al., 2018b). In the current QE approaches, black-box fea- tures are learned representations extracted by fine- tuning pre-trained multilingual or cross-lingual sen- tence encoders such as BERT (Devlin et al., 2018), ∗Equal contribution. XLM-R (Conneau et al., 2019) or LASER (Artetxe and Schwenk, 2019). These supervised approaches have led to the state-of-the-art (SOTA) results in this task (Kepler et al., 2019; Fonseca et al., 2019), similarly to what has been observed for a myriad of other downstream natural language processing applications that rely on cross-lingual sentence sim- ilarity. Glass-box features are usually obtained by extracting various types of information from the MT system, e.g. lexical probability or language model probability in the case of statistical MT sys- tems (Blatz et al., 2004b), or more recently softmax probability and attention weights from neural MT models (Fomicheva et al., 2020). Glass-box ap- proach is potentially useful for low resource or zero- shot scenarios as it does not require large amounts of labelled data for training, but it does not perform as well as SOTA supervised models. QE is therefore generally framed as a supervised machine learning problem, with models trained on data labelled for quality for each language pair. Training data publicly available …

Content truncated to 500 words