Learning Discriminative Projections for Text Similarity Measures

Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 247–256, Portland, Oregon, USA, 23–24 June 2011. c⃝2011 Association for Computational Linguistics Learning Discriminative Projections for Text Similarity Measures Wen-tau Yih Kristina Toutanova John C. Platt Christopher Meek Microsoft Research One Microsoft Way Redmond, WA 98052, USA {scottyih,kristout,jplatt,meek}@microsoft.com Abstract Traditional text similarity measures consider each term similar only to itself and do not model semantic relatedness of terms. We pro- pose a novel discriminative training method that projects the raw term vectors into a com- mon, low-dimensional vector space. Our ap- proach operates by finding the optimal matrix to minimize the loss of the pre-selected sim- ilarity function (e.g., cosine) of the projected vectors, and is able to efficiently handle a large number of training examples in the high- dimensional space. Evaluated on two very dif- ferent tasks, cross-lingual document retrieval and ad relevance measure, our method not only outperforms existing state-of-the-art ap- proaches, but also achieves high accuracy at low dimensions and is thus more efficient. 1 Introduction Measures of text similarity have many applications and have been studied extensively in both the NLP and IR communities. For example, a combination of corpus and knowledge based methods have been invented for judging word similarity (Lin, 1998; Agirre et al., 2009). Similarity derived from a large- scale Web corpus has been used for automatically extending lists of typed entities (Vyas and Pantel, 2009). Judging the degree of similarity between documents is also fundamental to classical IR prob- lems such as document retrieval (Manning et al., 2008). In all these applications, the vector-based similarity method is the most widely used. Term vectors are first constructed to represent the origi- nal text objects, where each term is associated with a weight indicating its importance. A pre-selected function operating on these vectors, such as cosine, is used to output the final similarity score. This ap- proach has not only proved to be effective, but is also efficient. For instance, only the term vectors rather than the raw data need to be stored. A pruned inverse index can be built to support fast similarity search. However, the main weakness of this term-vector representation is that different but semantically re- lated terms are not matched and cannot influence the final similarity score. As an illustrative ex- ample, suppose the two compared term-vectors are: {purchase:0.4, used:0.3, automobile:0.2} and {buy:0.3, pre-owned: 0.5, car: 0.4}. Even though the two vectors represent very similar concepts, their similarity score will be 0, for functions like cosine, overlap or Jaccard. Such an issue is more severe in cross-lingual settings. Because language vocab- ularies typically have little overlap, term-vector rep- resentations are completely inapplicable to measur- ing similarity between documents in different lan- guages. The general strategy to handle this prob- lem is to map the raw representation to a common concept space, where extensive approaches have been proposed. Existing methods roughly fall into three categories. Generative topic models like La- tent Dirichlet Allocation (LDA) (Blei et al., 2003) assume that …

Content truncated to 500 words