Penn: Using Word Similarities to better Estimate Sentence Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 679–683, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Penn: Using Word Similarities to better Estimate Sentence Similarity Sneha Jha and H. Andrew Schwartz and Lyle H. Ungar University of Pennsylvania Philadelphia, PA, USA {jhasneha, hansens, ungar}@seas.upenn.edu Abstract We present the Penn system for SemEval- 2012 Task 6, computing the degree of seman- tic equivalence between two sentences. We explore the contributions of different vector models for computing sentence and word sim- ilarity: Collobert and Weston embeddings as well as two novel approaches, namely eigen- words and selectors. These embeddings pro- vide different measures of distributional simi- larity between words, and their contexts. We used regression to combine the different simi- larity measures, and found that each provides partially independent predictive signal above baseline models. 1 Introduction We compute the semantic similarity between pairs of sentences by combining a set of similarity met- rics at various levels of depth, from surface word similarity to similarities derived from vector mod- els of word or sentence meaning. Regression is then used to determine optimal weightings of the differ- ent similarity measures. We use this setting to as- sess the contributions from several different word embeddings. Our system is based on similarities computed us- ing multiple sets of features: (a) naive lexical fea- tures, (b) similarity between vector representations of sentences, and (c) similarity between constituent words computed using WordNet, using the eigen- word vector representations of words , and using se- lectors, which generalize words to a set of words that appear in the same context. 2 System Description This section briefly describes the feature sets used to arrive at a similarity measure between sentences. We compare the use of word similarities based on three different embeddings for words neural embeddings using recursive autoencoders, eigenwords and selec- tors. 2.1 Neural Models of Word Representation An increasingly popular approach is to learn repre- sentational embeddings for words from a large col- lection of unlabeled data (typically using a genera- tive model), and to use these embeddings to augment the feature set of a supervised learner. These models are based on the distributional hypothesis in linguis- tics that words that occur in similar contexts tend to have similar meanings. The similarities between these vectors indicate similarity in the meanings of corresponding words. The state of the art model in paraphrase detection uses an unsupervised recursive autoencoder (RAE) model based on an unfolding objective that learn feature vectors for phrases in syntactic parse trees (Socher et al., 2011). The idea of neural language models is to jointly learn an embedding of words into an n-dimensional vector space that capture dis- tributional syntactic and semantic information via the words co-occurrence statistics. Further details and evaluations of these embeddings are discussed in Turian et al. (2010). Once the distributional syntactic and semantic matrix is learned on an unlabeled corpus, one can use it for subsequent tasks by using each words vec- tor to represent that …

Content truncated to 500 words