Semantic Textual Similarity for MT evaluation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 52–58, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Semantic Textual Similarity for MT evaluation Julio Castillo†‡ Paula Estrella‡ ‡FaMAF, UNC, Argentina †‡UTN-FRC, Argentina jotacastillo@gmail.com pestrella@famaf.unc.edu.ar Abstract This paper describes the system used for our participation in the WMT12 Machine Transla- tion evaluation shared task. We also present a new approach to Machine Translation evaluation based on the recently defined task Semantic Textual Similarity. This problem is addressed using a textual entail- ment engine entirely based on WordNet se- mantic features. We described results for the Spanish-English, Czech-English and German-English language pairs according to our submission on the Eight Workshop on Statistical Machine Translation. Our first experiments reports a competitive score to system level. 1 Introduction The evaluation of Machine Translation (MT) has become as important as MT itself over the last few years. This is evidenced by the fact that there are now specific forums to present and test new met- rics, such as the Workshop for Statistical MT (WMT) or the NIST MetricsMatr. Every year a vast number of MT metrics are created, the majori- ty being automatic, and seeking to find an efficient, low labor-intensive and reliable evaluation method as an alternative to human-based evaluation. Automatic metrics employ different evaluation strategies: classical MT automatic metrics, such as BLEU (Papineni et al., 2002), NIST (Doddington. 2002), WER (Tillmann et al., 1997), PER (Nießen et al., 2000) are language-independent based on n- gram matching (considering or not the ordering of words in a sentence); other use some kind of lan- guage-specific knowledge, for example METEOR (Banerjee et al., 2005), which uses WordNet to match synonyms if exact matchings do not occur, and METEOR-NEXT (Denkowski et al., 2010) that, in addition to METEOR’s features, incorpo- rates paraphrases; and more sophisticated metrics use deeper linguistic information, as for example the DCU-LFG metric (Yifan et al., 2010). However, relatively few attempts have been made to use semantic information for MT evalua- tion. Moreover, only one work has been published about using semantic equivalence (known as Tex- tual Entailment) of texts for MT evaluation. In this work we propose an improved metric, based on TE features, that indicates to what extent a candidate sentence is equivalent to a reference. The paper is organized as follows: Section 2 de- scribes the relevant work done on semantic orient- ed MT evaluation, Section 3 describes the architecture of the system to compute our metric, then Section 4 relates TE and semantic textual similarity to MT, and Section 5 presents some results obtained with our TE-based metric; and finally Section 6 summarize some conclusions and future work. 2 Related work Given the vast literature in the field of MT evalua- tion, in this section we briefly mention a few at- tempts to evaluate MT based on semantic features, which we deem most recent and important. 2.1 Semantics for MT evaluation Giménez and Márquez (2007) present a set of met- rics operating over shallow semantic structures, …

Content truncated to 500 words