Incorporate Semantic Structures into Machine Translation Evaluation via UCCA

Proceedings of the 5th Conference on Machine Translation (WMT), pages 934–939 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 934 Incorporate Semantic Structures into Machine Translation Evaluation via UCCA Jin Xu1, Yinuo Guo2, Junfeng Hu2 1Yuanpei College, Peking University 2Key Laboratory of Computational Linguistics, School of EECS, Peking University {jinxu, gyn0806, hujf}@pku.edu.cn Abstract Copying mechanism has been commonly used in neural paraphrasing networks and other text generation tasks, in which some important words in the input sequence are preserved in the output sequence. Similarly, in machine translation, we notice that there are certain words or phrases appearing in all good trans- lations of one source text, and these words tend to convey important semantic informa- tion. Therefore, in this work, we define words carrying important semantic meanings in sentences as semantic core words. More- over, we propose an MT evaluation approach named Semantically Weighted Sentence Simi- larity (SWSS). It leverages the power of UCCA to identify semantic core words, and then cal- culates sentence similarity scores on the over- lap of semantic core words. Experimental results show that SWSS can consistently im- prove the performance of popular MT evalua- tion metrics which are based on lexical similar- ity. 1 Introduction Machine Translation Evaluation (MTE) is to evalu- ate the quality of sentences produced by Machine Translation (MT) systems. Most automatic MT evaluation metrics compare the candidate sentences from MT systems with reference sentences from human translation to produce a similarity score, in contrast some other reference-less metrics directly compare candidate sentences and source sentences. According to the observation of well-translated sentences, we find out that there are certain words or phrases appearing in all good translations of one source text. This phenomenon is consistent with the intuition of copying mechanism (Gu et al., 2016), which has been widely used in lots of text generation tasks. In the field of MT evaluation, Meteor++ (Guo et al., 2018) firstly proposes the concept of copy knowledge to define the words with Word A Word B ... Word E Word F ... UCCA graph Semantic Core Words Within All Words Candidate Reference Word Match F1 Score Output Score Penalties Extract Extract Word A Word C ... Word D Word E ... Figure 1: An illustration of the process of SWSS. copy property, and it further incorporates the copy knowledge into Meteor (Denkowski and Lavie, 2014) to improve its performance. Specifically, it attempts to find copy words of references and candidate sentences, and uses the overlap of these words to modify the calculation of precision and recall of Meteor. However, Meteor++ uses named entities as an alternative to copy knowledge in its experiments, resulting in a limited range of selected copy words and a slight improvement. In this work, we argue that words undertaking important semantic meanings should be exactly expressed during the translation procedure, which we define as semantic core words. This concept is much more general and closer to linguistic intuition compared to the copy knowledge used in Meteor++. In order to …

Content truncated to 500 words