SRIUBC-Core: Multiword Soft Similarity Models for Textual Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 155–161, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics SRIUBC-Core: Multiword Soft Similarity Models for Textual Similarity Eric Yeh SRI International Menlo Park, CA USA yeh@ai.sri.com Eneko Agirre University of Basque Country Donostia, Basque Country e.agirre@ehu.es Abstract In this year’s Semantic Textual Similarity evaluation, we explore the contribution of models that provide soft similarity scores across spans of multiple words, over the pre- vious year’s system. To this end, we ex- plored the use of neural probabilistic language models and a TF-IDF weighted variant of Ex- plicit Semantic Analysis. The neural language model systems used vector representations of individual words, where these vectors were derived by training them against the context of words encountered, and thus reflect the dis- tributional characteristics of their usage. To generate a similarity score between spans, we experimented with using tiled vectors and Re- stricted Boltzmann Machines to identify simi- lar encodings. We find that these soft similar- ity methods generally outperformed our previ- ous year’s systems, albeit they did not perform as well in the overall rankings. A simple anal- ysis of the soft similarity resources over two word phrases is provided, and future areas of improvement are described. 1 Introduction For this year’s Semantic Textual Similarity (STS) evaluation, we built upon the best performing sys- tem we deployed last year with several methods for exploring the soft similarity between windows of words, instead of relying just on single token-to- token similarities. From the previous year’s eval- uation, we were impressed by the performance of features derived from bigrams and skip bigrams. Bi- grams capture the relationship between two concur- rent words, while skip bigrams can capture longer distance relationships. We found that characterizing the overlap in skip bigrams between the sentences in a STS problem pair proved to be a major contributor to last year’s system’s performance. Skip bigrams were matched on two criteria, lexi- cal matches, and via part of speech (POS). Lexical matching is brittle, and even if the match were made on lemmas, we lose the ability to match against syn- onyms. We could rely on the token-to-token simi- larity methods to account for these non-lexical sim- ilarities, but these do not account for sequence nor dependencies in the sentencees. Using POS based matching allows for a level of generalization, but at a much broader level. What we would like to have is a model that can capture these long distance re- lationships at a level that is less broad than POS matching, but allows for a soft similarity scoring be- tween words. In addition, the ability to encompass a larger window without having to manually insert skips would be desirable as well. To this end we decided to explore the use of neu- ral probabilistic language models (NLPM) for cap- turing this kind of behavior (Bengio et al., 2003). NLPMs represent individual words as real valued …

Content truncated to 500 words