DATScore: Evaluating Translation with Data Augmented Translations

Findings of the Association for Computational Linguistics: EACL 2023, pages 942–952 May 2-6, 2023 ©2023 Association for Computational Linguistics DATScore: Evaluating Translation with Data Augmented Translations Moussa Kamal Eddine1, Guokan Shang2, Michalis Vazirgiannis1,3 1École Polytechnique, 2Linagora, 3AUEB Abstract The rapid development of large pretrained lan- guage models has revolutionized not only the field of Natural Language Generation (NLG) but also its evaluation. Inspired by the recent work of BARTScore: a metric leveraging the BART language model to evaluate the qual- ity of generated text from various aspects, we introduce DATScore. DATScore uses data aug- mentation techniques to improve the evalua- tion of machine translation. Our main find- ing is that introducing data augmented transla- tions of the source and reference texts is greatly helpful in evaluating the quality of the gener- ated translation. We also propose two novel score averaging and term weighting strategies to improve the original score computing pro- cess of BARTScore. Experimental results on WMT show that DATScore correlates better with human meta-evaluations than the other re- cent state-of-the-art metrics, especially for low- resource languages. Ablation studies demon- strate the value added by our new scoring strate- gies. Moreover, we report in our extended ex- periments the performance of DATScore on 3 NLG tasks other than translation Code is pub- licly available1. 1 Introduction Massive pretrained language models have brought significant improvement to NLG tasks (Lewis et al., 2020). Recent systems can even generate texts of higher quality than human-annotated ones (Peyrard, 2019). At the same time, standard metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), for translation and summarization respec- tively, have not evolved for the past two decades (Bhandari et al., 2020). These metrics rely on sur- face lexicographic matches, making them partic- ularly unsuitable for evaluating modern systems operating with embeddings at the semantic level that often generate paraphrases (Ng and Abrecht, 1https://github.com/moussaKam/datscore 2015). To address this issue, many metrics have been proposed (Sai et al., 2022), but none of them were widely adopted until the release of BERTSore (Zhang et al., 2019) and MoverScore (Zhao et al., 2019). These metrics take advantage of large pre- trained language models like BERT (Devlin et al., 2019), which are now being used in nearly all NLP tasks (Qiu et al., 2020; Min et al., 2021). In this work, we focus on the task of evaluating machine translation. We propose an extension of BARTScore (Yuan et al., 2021), a recent metric ex- ploiting the BART seq2seq language model (Lewis et al., 2020) to evaluate the quality of generated text from various aspects. BARTScore covers four evaluation facets: Faithfulness, Precision, Recall, and F-score, derived from different generation di- rections between the source text, the hypothesis (the text generated by a system given the source), and the reference (the reference text for the gener- ation, often provided by human annotators). The scores are obtained by pairing the three entities dif- ferently at the input or the output side of a trained seq2seq model for fetching …

Content truncated to 500 words