Unsupervised Embedding-based Metric for MT Evaluation with Improved Human Correlation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 558 - 563 December 7-8, 2022 ©2022 Association for Computational Linguistics Unsupervised Embedding-based Metric for MT Evaluation with Improved Human Correlation Ananya Mukherjee and Manish Shrivastava Machine Translation - Natural Language Processing Lab Language Technologies Research Centre International Institute of Information Technology - Hyderabad ananya.mukherjee@research.iiit.ac.in m.shrivastava@iiit.ac.in Abstract In this paper, we describe our submission to the WMT22 metrics shared task. Our metric focuses on computing contextual and syntactic equivalences along with lexical, morphologi- cal and semantic similarity. The intent is to capture fluency and context of the MT outputs along with their adequacy. Fluency is captured using syntactic similarity and context is cap- tured using sentence similarity leveraging sen- tence embeddings. The final sentence trans- lation score is the weighted combination of three similarity scores: a) Syntactic Similarity b) Lexical, Morphological and Semantic Simi- larity and c) Contextual Similarity. This paper outlines two improved versions of MEE i.e., MEE2 and MEE4. Additionally, we perform our experiments on language pairs of en-de, en-ru and zh-en from WMT17-19 testset and further report the correlation with human as- sessments. Our submission will be made avail- able at https://github.com/AnanyaCoder/ WMT22Submission. 1 Introduction Neural Machine Translation (NMT) systems have emerged with an increased research interest in re- cent times and significantly enhanced the MT qual- ity. However, the MT research community still relies mainly on antiquated metrics and no new, universally adopted standard metric has emerged. In the last few years, research in Machine Transla- tion (MT) evaluation has made significant progress. A metrics-shared task is held annually at the WMT conference, where new evaluation metrics are pro- posed and those which correlates highly with human judgements are presented from the pool of newly defined metrics. Neural-based metrics largely dominated the last two years of the WMT Metrics Task (Freitag et al., 2021; Mathur et al., 2020; Ma et al., 2019). Nevertheless, n-gram based or lexical-based metrics remain popular as automatic MT evaluation metric due to their ag- ile and light-weighted nature. Traditionally, auto- matic metrics for evaluating MT quality have re- lied on estimating the similarity between machine outputs and reference sentences in the target lan- guage. However, advanced NMT methods yield high-quality translations that might have lexical, morphological, syntactic variations and different word choices having similar meanings. Typically, the machine output diverges from monotonic lex- ical transfer between the source and target lan- guages. Widely used evaluation metrics rely on basic, lexical-level features as they calculate the surface similarity between the hypothesis and refer- ence sentences by counting the number of matching n-grams (Papineni et al., 2002; Doddington, 2002). Metrics relying on n-gram overlap cannot appropri- ately capture morphological, syntactic and seman- tic variations as they are sensitive to only lexical variations. METEOR (Denkowski and Lavie, 2014; Gupta et al., 2010; Lavie and Denkowski, 2009; Lavie and Agarwal, 2007; Banerjee and Lavie, 2005) captures semantic variations but it is highly dependent on language specific tools . Hence, there is huge requirement for a …

Content truncated to 500 words