Improving Lexical Semantics for Sentential Semantics: Modeling Selectional Preference and Similar Words in a Latent Variable Model

Proceedings of NAACL-HLT 2013, pages 739–745, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Improving Lexical Semantics for Sentential Semantics: Modeling Selectional Preference and Similar Words in a Latent Variable Model Weiwei Guo Department of Computer Science Columbia University weiwei@cs.columbia.edu Mona Diab Department of Computer Science George Washington University mtdiab@gwu.edu Abstract Sentence Similarity [SS] computes a similar- ity score between two sentences. The SS task differs from document level semantics tasks in that it features the sparsity of words in a data unit, i.e. a sentence. Accordingly it is crucial to robustly model each word in a sen- tence to capture the complete semantic picture of the sentence. In this paper, we hypoth- esize that by better modeling lexical seman- tics we can obtain better sentential semantics. We incorporate both corpus-based (selectional preference information) and knowledge-based (similar words extracted in a dictionary) lex- ical semantics into a latent variable model. The experiments show state-of-the-art perfor- mance among unsupervised systems on two SS datasets. 1 Introduction Sentence Similarity [SS] is emerging as a crucial step in many NLP tasks that focus on sentence level semantics such as word sense disambiguation (Guo and Diab, 2010; Guo and Diab, 2012a), summariza- tion (Zhou et al., 2006), text coherence (Lapata and Barzilay, 2005), tweet clustering (Sankaranarayanan et al., 2009; Jin et al., 2011), etc. SS operates in a very small context, on average 11 words per sen- tence in Semeval-2012 dataset (Agirre et al., 2012), resulting in inadequate evidence to generalize to ro- bust sentential semantics. Weighted Textual Matrix Factorization [WTMF] (Guo and Diab, 2012b) is a latent variable model that outperforms Latent Semantic Analysis [LSA] (Deer- wester et al., 1990) and Latent Dirichelet Allocation [LDA] (Blei et al., 2003) models by a large margin in the SS task, yielding state-of-the-art performance on the LI06 (Li et al., 2006) SS dataset. However, all of these models make harsh simplifying assumptions on how a token is generated: (1) in LSA/WTMF, a token is generated by the inner product of the word latent vector and the document latent vector; (2) in LDA, all the tokens in a document are sampled from the same document level topic distribution. Under this framework, they ignore rich linguistic phenom- ena such as inter-word dependency, semantic scope of words, etc. This is a result of simply using docu- ment IDs as features to represent a word. Modeling quality lexical semantics in latent vari- able models does not draw enough attention in the community, since people usually apply dimension reduction techniques for documents, which have abundant words for extracting the document level semantics. However, in the SS setting, it is crucial to make good use of each word, given the limited num- ber of words in a sentence. We believe a reasonable word generation story will avoid introducing noise in sentential semantics, encouraging robust lexical semantics which can further boost the sentential se- mantics. In this paper, we explicitly encode lexical semantics, both corpus-based and knowledge-based information, in the WTMF …

Content truncated to 500 words