SSMT:A Machine Translation Evaluation View To Paragraph-to-Sentence Semantic Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 585–589, Dublin, Ireland, August 23-24, 2014. SSMT: A Machine Translation Evaluation View to Paragraph-to-Sentence Semantic Similarity Pingping Huang Department of Linguistic Engineering School of Software and Microelectronics Peking University, China girlhpp@163.com Baobao Chang Key Laboratory of Computational Linguistics, Ministry of Education Institute of Computational Linguistics Peking University, China chbb@pku.edu.cn Abstract This paper presents the system SSMT measuring the semantic similarity between a paragraph and a sentence submitted to the SemEval 2014 task3: Cross-level Se- mantic Similarity. The special difficulty of this task is the length disparity between the two semantic comparison texts. We adapt several machine translation evalua- tion metrics for features to cope with this difficulty, then train a regression model for the semantic similarity prediction. This system is straightforward in intuition and easy in implementation. Our best run gets 0.808 in Pearson correlation. METEOR- derived features are the most effective ones in our experiment. 1 Introduction Cross level semantic similarity measures the simi- larity between different levels of text unit, for ex- ample, between a document and a paragraph, or between a phrase and a word. Paragraph and sentence are the natural language units to convey opinions or state events in daily life. We can see posts on forums, questions and answers in Q&A communities and customer re- views on E-commerce websites, are mainly organ- ised in these two units. Better similarity measure- ment across them will be helpful in clustering sim- ilar answers or reviews. The paragraph-to-sentence semantic similarity subtask in SemEval2014 task3 (Jurgens et al., 2014) is the first semantic similarity competition across these two language levels. The special difficulty of this task is the length disparity be- tween the compared pair: a paragraph contains This work is licenced under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organizers. License details: http: //creativecommons.org/licenses/by/4.0/ 3.67 times the words of a sentence on average in the training set. Semantic similarity on different levels, for ex- ample, on word level (Mikolov et al., 2013), sen- tences level (B¨ar et al., 2012), document level (Turney and Pantel, 2010), have been well studied, yet methods on one level can hardly be applied to a different level, let alone be applied for the cross- level tasks. The work of Pilehvar et al.(2013) was an exception. They proposed a unified method for semantic comparison at multi-levels all the way from comparing word senses to comparing text documents Our work is inspired by automatic machine translation(MT) evaluation, in which different metrics are designed to compare the adequacy and fluency of a MT system’s output, called hypothe- sis, against a gold standard translation, called ref- erence. As MT evaluation metrics measure sen- tence pair similarity, it is a natural idea to general- ize them for paragraph-sentence pair. In this paper, we follow the motivations of sev- eral MT evaluation metrics yet made adaption to cope with the length disparity difficulty of this task, and combine …

Content truncated to 500 words