UNED: Improving Text Similarity Measures without Human Assessments

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 454–460, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics UNED: Evaluating Text Similarity Measures without Human Assessments∗ Enrique Amig´o † Julio Gonzalo † Jes´us Gim´enez ‡ Felisa Verdejo† † UNED, Madrid {enrique,julio,felisa}@lsi.uned.es ‡ Google, Dublin jesgim@gmail.com Abstract This paper describes the participation of UNED NLP group in the SEMEVAL 2012 Se- mantic Textual Similarity task. Our contribu- tion consists of an unsupervised method, Het- erogeneity Based Ranking (HBR), to combine similarity measures. Our runs focus on com- bining standard similarity measures for Ma- chine Translation. The Pearson correlation achieved is outperformed by other systems, due to the limitation of MT evaluation mea- sures in the context of this task. However, the combination of system outputs that partici- pated in the campaign produces three interest- ing results: (i) Combining all systems without considering any kind of human assessments achieve a similar performance than the best peers in all test corpora, (ii) combining the 40 less reliable peers in the evaluation campaign achieves similar results; and (iii) the correla- tion between peers and HBR predicts, with a 0.94 correlation, the performance of measures according to human assessments. 1 Introduction Imagine that we are interested in developing com- putable measures that estimate the semantic simi- larity between two sentences. This is the focus of the STS workshop in which this paper is presented. In order to optimize the approaches, the organizers ∗This work has been partially funded by the Madrid gov- ernment, grant MA2VICMR (S-2009/TIC- 1542), the Span- ish government, grant Holopedia (TIN2010-21128-C02-01) and the European Community’s Seventh Framework Programme (FP7/ 2007-2013) under grant agreement nr. 288024 (LiMo- SINe project). provide a training corpus with human assessments. The participants must improve their approaches and select three runs to participate. Unfortunately, we can not ensure that systems will behave similarly in both the training and test corpora. For instance, some Pearson correlations between system achieve- ments across test corpora in this competition are: 0.61 (MSRpar-MSRvid), 0.34 (MSRvid-SMTeur), or 0.49 (MSRpar-SMTeur). Therefore, we cannot expect a high correlation between the system per- formance in a specific corpus and the test corpora employed in the competition. Now, imagine that we have a magic box that, given a set of similarity measures, is able to pre- dict which measures will obtain the highest corre- lation with human assessments without actually re- quiring those assessments. For instance, suppose that putting all system outputs in the magic box, we obtain a 0.94 Pearson correlation between the pre- diction and the system achievements according to human assessments, as in Figure 1. The horizontal axis represents the magic box ouput, and the vertical axis represents the achievement in the competition. Each dot represents one system. In this case, we could decide which system or system combination to employ for a certain test set. Is there something like this magic box? The answer is yes. Indeed, what Figure 1 shows is the results of an unsupervised method to combine measures, …

Content truncated to 500 words