Learning Text Pair Similarity with Context-sensitive Autoencoders

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Learning Text Pair Similarity with Context-sensitive Autoencoders Hadi Amiri1, Philip Resnik1, Jordan Boyd-Graber2, Hal Daum´e III1 1Institute for Advanced Computer Studies University of Maryland, College Park, MD 2Department of Computer Science University of Colorado, Boulder, CO {hadi,resnik,hal}@umd.edu,jordan.boyd.graber@colorado.edu Abstract We present a pairwise context-sensitive Autoencoder for computing text pair sim- ilarity. Our model encodes input text into context-sensitive representations and uses them to compute similarity between text pairs. Our model outperforms the state-of-the-art models in two semantic re- trieval tasks and a contextual word simi- larity task. For retrieval, our unsupervised approach that merely ranks inputs with re- spect to the cosine similarity between their hidden representations shows comparable performance with the state-of-the-art su- pervised models and in some cases outper- forms them. 1 Introduction Representation learning algorithms learn repre- sentations that reveal intrinsic low-dimensional structure in data (Bengio et al., 2013). Such rep- resentations can be used to induce similarity be- tween textual contents by computing similarity be- tween their respective vectors (Huang et al., 2012; Silberer and Lapata, 2014). Recent research has made substantial progress on semantic similarity using neural networks (Rothe and Sch¨utze, 2015; Dos Santos et al., 2015; Severyn and Moschitti, 2015). In this work, we focus our attention on deep autoencoders and extend these models to integrate sentential or document context information about their inputs. We represent context information as low dimensional vectors that will be injected to deep autoencoders. To the best of our knowledge, this is the first work that enables integrating context into autoencoders. In representation learning, context may appear in various forms. For example, the context of a current sentence in a document could be ei- ther its neighboring sentences (Lin et al., 2015; Wang and Cho, 2015), topics associated with the sentence (Mikolov and Zweig, 2012; Le and Mikolov, 2014), the document that contains the sentence (Huang et al., 2012), as well as their com- binations (Ji et al., 2016). It is important to inte- grate context into neural networks because these models are often trained with only local informa- tion about their individual inputs. For example, recurrent and recursive neural networks only use local information about previously seen words in a sentence to predict the next word or composition.1 On the other hand, context information (such as topical information) often capture global informa- tion that can guide neural networks to generate more accurate representations. We investigate the utility of context informa- tion in three semantic similarity tasks: contextual word sense similarity in which we aim to predict semantic similarity between given word pairs in their sentential context (Huang et al., 2012; Rothe and Sch¨utze, 2015), question ranking in which we aim to retrieve semantically equivalent questions with respect to a given test question (Dos Santos et al., 2015), and answer ranking in which we aim to rank single-sentence answers with respect to a given question (Severyn …

Content truncated to 500 words