IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14210–14228 July 9-14, 2023 ©2023 Association for Computational Linguistics IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages Ananya B. Sai1 Tanay Dixit1 Vignesh Nagarajan2 Anoop Kunchukuttan2,4 Pratyush Kumar2,4 Mitesh M. Khapra1,2 Raj Dabre3 Indian Institute of Technology Madras1 AI4Bharat2 National Institute of Information and Communications Technology3 Microsoft4 ananya@cse.iitm.ac.in vignesh.vn.nagarajan@gmail.comdixittanay@gmail.com ankunchu@microsoft.com pratykumar@microsoft.com miteshk@cse.iitm.ac.in raj.dabre@nict.go.jp Abstract The rapid growth of machine translation (MT) systems necessitates meta-evaluations of evalu- ation metrics to enable selection of those that best reflect MT quality. Unfortunately, most meta-evaluation studies focus on European lan- guages, the observations for which may not always apply to other languages. Indian lan- guages, having over a billion speakers, are lin- guistically different from them, and to date, there are no such systematic studies focused solely on English to Indian language MT. This paper fills this gap through a Multidimensional Quality Metric (MQM) dataset consisting of 7000 fine-grained annotations, spanning 5 In- dian languages and 7 MT systems. We evaluate 16 metrics and show that, pre-trained metrics like COMET have the highest correlations with annotator scores as opposed to n-gram metrics like BLEU. We further leverage our MQM an- notations to develop an Indic-COMET metric and show that it outperforms COMET coun- terparts in both human scores correlations and robustness scores in Indian languages. Addi- tionally, we show that the Indic-COMET can outperform COMET on some unseen Indian languages. We hope that our dataset and analy- sis will facilitate further research in Indic MT evaluation. 1 Introduction Natural language generation (NLG) has seen rapid progress in the past few years due to advance- ments in the field of large language models (LLMs) (Lewis et al., 2020; Liu et al., 2020; Dabre et al., 2022; Scao et al., 2022). Although initial research had focused on high-resource languages, recently the focus has shifted to middle-resource and low- resource languages. In the context of machine translation (MT), there is increasing interest in building massively multilingual models support- ing numerous translation directions. For example, Costa-jussà et al. (2022) release a model which supports around 200 languages (40K directions). While this is commendable, to make MT truly in- clusive, it is important that various design choices in the MT life-cycle are evaluated for low-resource languages and not simply transferred and adapted from English. One such important choice is of the correct evaluation metric to be used for evaluating MT systems. A recent survey by Sai et al. (2022) has shown that over the last decade, many evaluation metrics have been proposed for MT. In parallel, several works (Callison-Burch et al., 2006; Sai et al., 2021; Mathur et al., 2020a; Tan et al., 2015; Fabbri et al., 2021) have shown the inadequacy of popular met- rics, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004). However, many languages are not rep- resented in these works and most of the focus is on European languages. On the other …

Content truncated to 500 words