REUSE: REference-free UnSupervised Quality Estimation Metric

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 564 - 568 December 7-8, 2022 ©2022 Association for Computational Linguistics REUSE: REference-free UnSupervised quality Estimation Metric Ananya Mukherjee and Manish Shrivastava Machine Translation - Natural Language Processing Lab Language Technologies Research Centre International Institute of Information Technology - Hyderabad ananya.mukherjee@research.iiit.ac.in m.shrivastava@iiit.ac.in Abstract This paper describes our submission to the WMT2022 shared metrics task. Our unsuper- vised metric estimates the translation quality at chunk-level and sentence-level. Source and target sentence chunks are retrieved by using a multi-lingual chunker. Chunk-level similar- ity is computed by leveraging BERT contex- tual word embeddings and sentence similarity scores are calculated by leveraging sentence embeddings of Language-Agnostic BERT mod- els. The final quality estimation score is ob- tained by mean pooling the chunk-level and sentence-level similarity scores. This paper out- lines our experiments and also reports the corre- lation with human judgements for en-de, en-ru and zh-en language pairs of WMT17, WMT18 and WMT19 testsets. Our submission will be made available at https://github.com/ AnanyaCoder/WMT22Submission_REUSE 1 Introduction Quality Estimation (QE) is an essential component of the machine translation workflow as it assesses the quality of the translated output without con- ferring reference translations (Specia et al., 2009; Blatz et al., 2004). High quality reference transla- tions are often hard to find, QE helps to evaluate the translation quality based on the source sentences. Recently QE has emerged as an alternative eval- uation approach for NMT systems (Specia et al., 2018). Recently, many researchers have been work- ing on QE, as a part of Quality Estimation Shared Task, several QE systems (Zerva et al., 2021; Lim et al., 2021; Chowdhury et al., 2021; Geigle et al., 2021) were evaluated in WMT conference (Bar- rault et al., 2021). However, most of the quality estimation systems are supervised i.e., the model re- gresses on the human judgements. Often, human as- sessments are not available and it is very difficult to procure high quality human judgements. This moti- vated our research to emerge with an Unsupervised Quality Estimation System. Also, QE is usually performed at different granularity (e.g., word, sen- tence, document) (Kepler et al., 2019); in this work, we focus on the chunk-level and sentence-level sim- ilarity. The final QE score of the target sentence is obtained by mean pooling the chunk similarity scores and sentence similarity scores. Overall, our main contribution is as follows: • We propose a concept of chunk level similarity i.e., matching the source and target chunks by leveraging multilingual BERT embeddings. • We release a multilingual chunking model which returns meaningful word group bound- aries. • We present our unsupervised reference free QE metric (REUSE) that estimates the qual- ity of translation by doing a chunk-level and sentence-level comparison with the source. 1.1 Motivation to use chunks Usually, the words in translated output might not always follow the word sequence of the source text. However, it is observed that few word-groups often occur together irrespective of the order in source. Figure 1 illustrates two …

Content truncated to 500 words