Learning the Impact of Machine Translation Evaluation Metrics for Semantic Textual Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 398–403, Hissar, Bulgaria, Sep 7–9 2015. Learning the Impact of Machine Translation Evaluation Metrics for Semantic Textual Similarity Simone Magnolini University of Brescia, Fondazione Bruno Kessler Trento, Italy magnolini@fbk.eu Ngoc Phuoc An Vo University of Trento, Fondazione Bruno Kessler Trento, Italy ngoc@fbk.eu Octavian Popescu IBM Research, T.J. Watson Yorktown, US o.popescu@us.ibm.com Abstract We present a work to evaluate the hy- pothesis that automatic evaluation metrics developed for Machine Translation (MT) systems have significant impact on pre- dicting semantic similarity scores in Se- mantic Textual Similarity (STS) task for English, in light of their usage for para- phrase identification. We show that differ- ent metrics may have different behaviors and significance along the semantic scale [0-5] of the STS task. In addition, we com- pare several classification algorithms us- ing a combination of different MT metrics to build an STS system; consequently, we show that although this approach obtains state of the art result in paraphrase iden- tification task, it is insufficient to achieve the same result in STS. 1 Introduction Semantic related tasks have become a noticed trend in Natural Language Processing (NLP) com- munity. Particularly, the Semantic Textual Simi- larity (STS) task has captured a huge attention in the NLP community despite being recently intro- duced since SemEval 2012 and continuing in Se- mEval 2013, 2014 and 2015 (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). Basically, the task requires to build sys- tems which can compute the similarity degree be- tween two given sentences. The similarity degree is scaled as a real score from 0 (no relevance) to 5 (semantic equivalence). The evaluation is done by computing the correlation between human judg- ment scores and systems’ predictions by the mean of Pearson correlation method. In contrast, Machine Translation evaluation metrics are designed to assess if the output of a MT system is semantically equivalent to a set of reference translations. In SemEval 2012, the sys- tem made by (de Souza et al., 2012) and then the system (Barrón Cedeño et al., 2013) in SemEval 2013 introduced the approach of using a set of MT evaluation metrics together with other lexi- cal and syntactic features to predict the semantic similarity scores in STS. Although this approach shows promising results, there was no in-depth analysis on the impact of the evaluation metrics to the overall performance and how each metric be- haves on STS data. Moreover, as being inspired by the literature (Madnani et al., 2012) for para- phrase recognition, which obtains the state of art result on the Microsoft Research paraphrase cor- pus (MSRP) (Dolan et al., 2004), we decide to an- alyze the impact of MT evaluation metrics in STS. Our aim consists of two folds, (1) to obtain a clear idea of how each individual metric behaves and correlates with the human-judgement seman- tic similarity, and (2) to examine the approach of combining a set of chosen metrics to build …

Content truncated to 500 words