Learning Distributed Representations of Texts and Entities from Knowledge Base

Learning Distributed Representations of Texts and Entities from Knowledge Base Ikuya Yamada1,4 Hiroyuki Shindo2 Hideaki Takeda3 Yoshiyasu Takefuji4 ikuya@ousia.jp shindo@is.naist.jp takeda@nii.ac.jp takefuji@sfc.keio.ac.jp 1 Studio Ousia, Japan, 2 Nara Institute of Science and Technology, Japan 3 National Institute of Informatics, Japan, 4 Keio University, Japan Abstract We describe a neural network model that jointly learns distributed representations of texts and knowledge base (KB) entities. Given a text in the KB, we train our proposed model to predict entities that are relevant to the text. Our model is designed to be generic with the ability to address various NLP tasks with ease. We train the model using a large cor- pus of texts and their entity annotations ex- tracted from Wikipedia. We evaluated the model on three important NLP tasks (i.e., sen- tence textual similarity, entity linking, and fac- toid question answering) involving both unsu- pervised and supervised settings. As a result, we achieved state-of-the-art results on all three of these tasks. Our code and trained models are publicly available for further academic re- search.1 1 Introduction Methods capable of learning distributed representa- tions of arbitrary-length texts (i.e., fixed-length con- tinuous vectors that encode the semantics of texts), such as sentences and paragraphs, have recently attracted considerable attention (Le and Mikolov, 2014; Kiros et al., 2015; Li et al., 2015; Wieting et al., 2016; Hill et al., 2016b; Kenter et al., 2016). These methods aim to learn generic representations that are useful across domains similar to word em- bedding methods such as Word2vec (Mikolov et al., 2013b) and GloVe (Pennington et al., 2014). Another interesting approach is learning dis- tributed representations of entities in a knowledge 1https://github.com/studio-ousia/ntee base (KB) such as Wikipedia and Freebase. These methods encode information of entities in the KB into a continuous vector space. They are shown to be effective for various KB-related tasks such as en- tity search (Hu et al., 2015), entity linking (Hu et al., 2015; Yamada et al., 2016), and link prediction (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015). In this paper, we describe a novel method to bridge these two different approaches. In particular, we propose Neural Text-Entity Encoder (NTEE), a neural network model to jointly learn distributed representations of texts (i.e., sentences and para- graphs) and KB entities. For every text in the KB, our model aims to predict its relevant entities, and places the text and the relevant entities close to each other in a continuous vector space. We use human- edited entity annotations obtained from Wikipedia (see Table 1) as supervised data of relevant entities to the texts containing these annotations.2 Note that, KB entities have been conventionally used to model semantics of texts. A representa- tive example is Explicit Semantic Analysis (ESA) (Gabrilovich and Markovitch, 2007), which repre- sents the semantics of a text using a sparse vector space, where each dimension corresponds to the rel- evance score of the text to each entity. Essentially, ESA shows that text can be accurately represented using …

Content truncated to 500 words