UniMelb_NLP-CORE: Integrating predictions from multiple domains and feature sets for estimating semantic textual similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 207–215, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics UniMelb NLP-CORE: Integrating predictions from multiple domains and feature sets for estimating semantic textual similarity Spandana Gella,♣ Bahar Salehi,♠♣ Marco Lui,♠♣ Karl Grieser,♣ Paul Cook,♣ and Timothy Baldwin,♠♣ ♠ NICTA Victoria Research Laboratory ♣ Department of Computing and Information Systems, The University of Melbourne sgella@student.unimelb.edu.au, bsalehi@student.unimelb.edu.au mhlui@unimelb.edu.au, kgrieser@student.unimelb.edu.au paulcook@unimelb.edu.au, tb@ldwin.net Abstract In this paper we present our systems for cal- culating the degree of semantic similarity be- tween two texts that we submitted to the Se- mantic Textual Similarity task at SemEval- 2013. Our systems predict similarity using a regression over features based on the fol- lowing sources of information: string similar- ity, topic distributions of the texts based on latent Dirichlet allocation, and similarity be- tween the documents returned by an informa- tion retrieval engine when the target texts are used as queries. We also explore methods for integrating predictions using different training datasets and feature sets. Our best system was ranked 17th out of 89 participating systems. In our post-task analysis, we identify simple changes to our system that further improve our results. 1 Introduction Semantic Textual Similarity (STS) measures the de- gree of semantic similarity or equivalence between a pair of short texts. STS is related to many natural language processing applications such as text sum- marisation (Aliguliyev, 2009), machine translation, word sense disambiguation, and question answering (De Boni and Manandhar, 2003; Jeon et al., 2005). Two short texts are considered similar if they both convey similar messages. Often it is the case that similar texts will have a high degree of lexical over- lap, although this isn’t always so. For example, SC dismissed government’s review plea in Vodafone tax case and SC dismisses govt’s review petition on Vodafone tax verdict are semantically similar. These texts have matches in terms of exact words (SC, Vodafone, tax), morphologically-related words (dis- missed and dismisses), and abbreviations (govern- ment’s and govt’s). However, the usages (senses) of plea and petition, and case and verdict are also sim- ilar. One straightforward way of estimating semantic similarity of two texts is by using approaches based on the similarity of the surface forms of the words they contain. However, such methods are not capa- ble of capturing similarity or relatedness at the lexi- cal level, and moreover, they do not exploit the con- text in which individual words are used in a target text. Nevertheless, a variety of knowledge sources — including part-of-speech, collocations, syntax, and domain — can be used to identify the usage or sense of words in context (McRoy, 1992; Agirre and Martinez, 2001; Agirre and Stevenson, 2006) to ad- dress these issues. Despite their limitations, string similarity mea- sures have been widely used in previous seman- tic similarity tasks (Agirre et al., 2012; Islam and Inkpen, 2008). Latent variable models have also been used to estimate the semantic …

Content truncated to 500 words