MEANT 2.0: Accurate semantic MT evaluation for any output language

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 589–597 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics MEANT 2.0: Accurate semantic MT evaluation for any output language Chi-kiu Lo NRC-CNRC Multilingual Text Processing National Research Council Canada 1200 Montreal Road, Ottawa, ON K1A 0R6, Canada chikiu.lo@nrc-cnrc.gc.ca Abstract We describe a new version of MEANT, which participated in the metrics task of the Second Conference on Machine Trans- lation (WMT 2017). MEANT 2.0 uses idf- weighted distributional ngram accuracy to determine the phrasal similarity of seman- tic role fillers and yields better correlations with human judgments of translation qual- ity than earlier versions. The improved phrasal similarity enables a subversion of MEANT to accurately evaluate translation adequacy for any output language, even languages without an automatic semantic parser. Our results show that MEANT, which is a non-ensemble and untrained metric, consistently performs as well as the top participants in previous years - including ensemble and trained ones - across different output languages. We also present the timing statistics for MEANT for better estimation of the evaluation cost. MEANT 2.0 is open source and publicly available.1 1 Introduction We introduce a new version of MEANT, which participated in evaluating MT systems for all lan- guage pairs in the metrics task of the Second Conference on Machine Translation (WMT 2017). MEANT 2.0 is a non-ensemble and untrained met- ric that only requires a monolingual corpus in the output language to build the word embeddings and an automatic shallow semantic parser to obtain the predicate-argument structure to evaluate MT sys- tems for a language pair. We have also build a degraded subversion, MEANT 2.0 - nosrl, to eval- uate MT systems for any output language by re- 1http://chikiu-jackie-lo.org/home/index.php/meant moving the dependency on semantic parsers for semantic role labeling (SRL) the reference and the machine translations. The correlation of MEANT with human judgments has been improved by us- ing both inverse document frequency (idf) and dis- tributional ngram accuracy within the phrasal sim- ilarity calculation: the former to weight the impor- tance of each word for better adequacy, the latter to to account for word reordering for greater flu- ency. Our results show that MEANT consistently performs as well as the top participants in previous years across different output languages, including ensemble and trained participants. We also present the timing statistics that show the relatively low cost of running MEANT. This highly portable and open source semantic MT evaluation metric is a more accurate alternative to BLEU in evaluating translation quality for low-resource languages. 2 The family of MEANT MEANT and its variants (Lo et al., 2015, 2014; Lo and Wu, 2011a) evaluate translation adequacy by measuring the similarity of the semantic frames and their role fillers between the human reference and machine translations. Figure 1 illustrates the concept of MEANT - the semantic roles and their fillers of the reference translation match more with those of the MT2 than with those of the MT1, therefore MT2 …

Content truncated to 500 words