SentSim: Crosslingual Semantic Evaluation of Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3143–3156 June 6–11, 2021. ©2021 Association for Computational Linguistics 3143 SentSim: Crosslingual Semantic Evaluation of Machine Translation Yurun Song∗ Imperial College London UK yurun.song19@imperial.ac.uk Junchen Zhao∗ University of California, Irvine CA, US junchez3@uci.edu Lucia Specia Imperial College London UK l.specia@imperial.ac.uk Abstract Machine translation (MT) is currently evalu- ated in one of two ways: in a monolingual fashion, by comparison with the system out- put to one or more human reference trans- lations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative SentSim: relying on strong pretrained mul- tilingual word and sentence representations, we directly compare the source with the ma- chine translated sentence, thus avoiding the need for both reference translations and la- belled training data. The metric builds on state-of-the-art embedding-based approaches – namely BERTScore and Word Mover’s Dis- tance – by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on differ- ent datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model in- formation. 1 Introduction Automatically evaluating machine translation (MT) as well as other language generation tasks has been investigated for decades, with substantial progress in recent years due to the advances of pretrained contextual word embeddings. The general goal of such evaluation metrics is to estimate the semantic equivalence between the input text (e.g. a source sentence or a document) and an output text that has been modified in some way (e.g. a translation or summary), as well as the general quality of the output (e.g. fluency). As such, by definition met- rics should perform some forms of input-output comparisons. *Contributed equally to this work. However, this direct comparison has been proven hard in the past because of the natural differences between the two versions (such as different lan- guages). Instead, evaluation metrics have resorted to comparison against one or more correct outputs produced by humans, a.k.a. reference texts, where comparisons at the string level are possible and straightforward. A multitude of evaluation metrics have been proposed following this approach, espe- cially for MT, the application we focus on in this paper. These include the famous BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) for machine translation, ROUGE (Lin, 2004) for summarization, and CIDER (Vedantam et al., 2014) for image captioning. These traditional met- rics are based on simple-word, n-gram matching mechanisms or slight relaxations of these (e.g. syn- onyms) which are computationally efficient, but suffer from various limitations. In order to overcome the drawbacks of the tra- ditional string-based evaluation metrics, recent work (Williams et al., 2018; Bowman et al., …

Content truncated to 500 words