NaCTeM at SemEval-2016 Task 1: Inferring sentence-level semantic similarity from an ensemble of complementary lexical and sentence-level features

Proceedings of SemEval-2016, pages 614–620, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics NaCTeM at SemEval-2016 Task 1: Inferring sentence-level semantic similarity from an ensemble of complementary lexical and sentence-level features Piotr Przybyła, Nhung T. H. Nguyen, Matthew Shardlow, Georgios Kontonatsios and Sophia Ananiadou National Centre for Text Mining University of Manchester Manchester, UK {piotr.przybyla,nhung.nguyen,matthew.shardlow, georgios.kontonatsios,sophia.ananiadou}@manchester.ac.uk Abstract We present a description of the system submit- ted to the Semantic Textual Similarity (STS) shared task at SemEval 2016. The task is to assess the degree to which two sentences carry the same meaning. We have designed two different methods to automatically com- pute a similarity score between sentences. The first method combines a variety of semantic similarity measures as features in a machine learning model. In our second approach, we employ training data from the Interpretable Similarity subtask to create a combined word- similarity measure and assess the importance of both aligned and unaligned words. Finally, we combine the two methods into a single hy- brid model. Our best-performing run attains a score of 0.7732 on the 2015 STS evaluation data and 0.7488 on the 2016 STS evaluation data. 1 Introduction If you ask a computer if two sentences are identical, it will return an accurate decision in a split-second. Ask it to do the same for a million sentence pairs and it will take a few seconds — far quicker than any human. But similarity has many dimensions. Ask a computer if two sentences mean the same and it will stall, yet the human can answer instantly. Now we have the edge. But what metrics do we use in our personal cognitive similarity-scoring systems? The answer is at the heart of the semantic similarity task. In our solution, we have incorporated several cat- egories of features from a variety of sources. Our approach covers both low-level visual features such as length and edit-distance as well as high-level se- mantic features such as topic-models and alignment quality measures. Throughout our approach, we have found it easier to consider similarity at the lexical level. This is un- surprising, as each lexeme may be seen as a seman- tic unit with the sentence’s meaning built directly from a specific combination of lexemes. We have explored several methods for abstracting our lexical- level features to the sentence level as explained in Section 4. Our methods are built from both intuitive features as well as the results of error-analyses on our trial runs. We have combined multiple feature sources to build a powerful semantic-similarity clas- sifier, discarding non-performing features as neces- sary. 2 Related Work The presented system has been submitted to the Se- mantic Textual Similarity (STS) shared task at Se- mEval 2016, which has been organised since 2012 (Agirre et al., 2016). Existing approaches to the problem adopt a plethora of similarity measures in- cluding string-based, content-based and knowledge- based methods. String-based methods (B¨ar et al., 2012; Malakasiotis and Androutsopoulos, 2007; Jimenez et al., 2012) exploit surface features (e.g., character …

Content truncated to 500 words