UPC-CORE: What Can Machine Translation Evaluation Metrics and Wikipedia Do for Estimating Semantic Textual Similarity?

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 143–147, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics UPC-CORE: What Can Machine Translation Evaluation Metrics and Wikipedia Do for Estimating Semantic Textual Similarity? Alberto Barr´on-Cede˜no1,2 Llu´ıs M`arquez1 Maria Fuentes1 Horacio Rodr´ıguez1 Jordi Turmo1 1 TALP Research Center, Universitat Polit`ecnica de Catalunya Jordi Girona Salgado 1–3, 08034, Barcelona, Spain 2 Facultad de Inform´atica, Universidad Polit´ecnica de Madrid Boadilla del Monte, 28660 Madrid, Spain albarron, lluism, mfuentes, horacio, turmo @lsi.upc.edu Abstract In this paper we discuss our participation to the 2013 Semeval Semantic Textual Similarity task. Our core features include (i) a set of met- rics borrowed from automatic machine trans- lation, originally intended to evaluate auto- matic against reference translations and (ii) an instance of explicit semantic analysis, built upon opening paragraphs of Wikipedia 2010 articles. Our similarity estimator relies on a support vector regressor with RBF kernel. Our best approach required 13 machine transla- tion metrics + explicit semantic analysis and ranked 65 in the competition. Our post- competition analysis shows that the features have a good expression level, but overfitting and —mainly— normalization issues caused our correlation values to decrease. 1 Introduction Our participation to the 2013 Semantic Textual Sim- ilarity task (STS) (Agirre et al., 2013)1 was focused on the CORE problem: GIVEN TWO SENTENCES, s1 AND s2, QUANTIFIABLY INFORM ON HOW SIMI- LAR s1 AND s2 ARE. We considered real-valued fea- tures from four different sources: (i) a set of linguis- tic measures computed with the Asiya Toolkit for Automatic MT Evaluation (Gim´enez and M`arquez, 2010b), (ii) an instance of explicit semantic analy- sis (Gabrilovich and Markovitch, 2007), built on top of Wikipedia articles, (iii) a dataset predictor, and (iv) a subset of the features available in Takelab’s Semantic Text Similarity system (ˇSari´c et al., 2012). 1http://ixa2.si.ehu.es/sts/ Our approaches obtained an overall modest result compared to other participants (best position: 65 out of 89). Nevertheless, our post-competition analysis shows that the low correlation was caused mainly by a deficient data normalization strategy. The paper distribution is as follows. Section 2 of- fers a brief overview of the task. Section 3 describes our approach. Section 4 discuss our experiments and obtained results. Section 5 provides conclusions. 2 Task Overview Detecting two similar text fragments is a difficult task in cases where the similarity occurs at seman- tic level, independently of the implied lexicon (e.g in cases of dense paraphrasing). As a result, simi- larity estimation models must involve features other than surface aspects. The STS task is proposed as a challenge focused in short English texts of dif- ferent nature: from automatic machine translation alternatives to human descriptions of short videos. The test partition also included texts extracted from news headlines and FrameNet–Wordnet pairs. The range of similarity was defined between 0 (no relation) up to 5 (semantic equivalence). The gold standard values were averaged from different human-made annotations. The expected system’s output was …

Content truncated to 500 words