CSE: Conceptual Sentence Embeddings based on Attention Model

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 505–515, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics CSE: Conceptual Sentence Embeddings based on Attention Model Yashen Wang, Heyan Huang∗, Chong Feng, Qiang Zhou, Jiahui Gu and Xiong Gao Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, School of Computer, Beijing Institute of Technology, Beijing, P. R. China {yswang,hhy63,fengchong,qzhou,gujh,gaoxiong}@bit.edu.cn Abstract Most sentence embedding models typical- ly represent each sentence only using word surface, which makes these models indis- criminative for ubiquitous homonymy and polysemy. In order to enhance represen- tation capability of sentence, we employ conceptualization model to assign associ- ated concepts for each sentence in the tex- t corpus, and then learn conceptual sen- tence embedding (CSE). Hence, this se- mantic representation is more expressive than some widely-used text representation models such as latent topic model, espe- cially for short-text. Moreover, we fur- ther extend CSE models by utilizing a lo- cal attention-based model that select rel- evant words within the context to make more efficient prediction. In the experi- ments, we evaluate the CSE models on two tasks, text classification and information retrieval. The experimental results show that the proposed models outperform typi- cal sentence embed-ding models. 1 Introduction Many natural language processing applications re- quire the input text to be represented as a fixed- length feature, of which sentence representation is very important. Perhaps the most common fixed- length vector representation for texts is the bag-of- words or bag-of-n-grams (Harris, 1970). Howev- er, they suffer severely from data sparsity and high dimensionality, and have very little sense about the semantics of words or the distances between the words. Recently, in sentence representation and classification, deep neural network (DNN) ap- proaches have achieved state-of-the-art results (Le ∗ The contact author. and Mikolov, 2014; Liu et al., 2015; Palangi et al., 2015; Wieting et al., 2015). Despite of their use- fulness, recent sentence embeddings face several challenges: (i) Most sentence embedding models represent each sentence only using word surface, which makes these models indiscriminative for u- biquitous polysemy; (ii) For short-text, however, neither parsing nor topic modeling works well be- cause there are simply not enough signals in the input; (iii) Setting window size of context word- s is very difficult. To solve these problems, we must derive more semantic signals from the input sentence, e.g., concepts. Besides, we should as- signed different attention for different contextual word, to enhance the influence of words that are relevant for each prediction. This paper proposed Conceptual Sentence Em- bedding (CSE), an unsupervised framework that learns continuous distributed vector representa- tions for sentence. Specially, by innovatively in- troducing concept information, this concept-level vector representations of sentence are learned to predict the surrounding words or target word in contexts. Our research is inspired by the recent work in learning vector representations of word- s using deep learning strategy (Mikolov et al., 2013a; Le and Mikolov, 2014). More precisely, we first obtain …

Content truncated to 500 words