Neural Attentive Bag-of-Entities Model for Text Classification

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 563–573 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 563 Neural Attentive Bag-of-Entities Model for Text Classification Ikuya Yamada1,3 ikuya@ousia.jp Hiroyuki Shindo2,3 shindo@is.naist.jp 1Studio Ousia, Tokyo, Japan 2Nara Institute of Science and Technology, Nara, Japan 3RIKEN AIP, Tokyo, Japan Abstract This study proposes a Neural Attentive Bag- of-Entities model, which is a neural network model that performs text classification using entities in a knowledge base. Entities pro- vide unambiguous and relevant semantic sig- nals that are beneficial for capturing seman- tics in texts. We combine simple high-recall entity detection based on a dictionary, to de- tect entities in a document, with a novel neu- ral attention mechanism that enables the model to focus on a small number of unambigu- ous and relevant entities. We tested the ef- fectiveness of our model using two standard text classification datasets (i.e., the 20 News- groups and R8 datasets) and a popular factoid question answering dataset based on a trivia quiz game. As a result, our model achieved state-of-the-art results on all datasets. The source code of the proposed model is avail- able online at https://github.com/ wikipedia2vec/wikipedia2vec. 1 Introduction Text classification is an important task, and its applications span a wide range of activities such as topic classification, spam detection, and sen- timent classification. Recent studies showed that models based on neural networks can outperform conventional models (e.g., na¨ıve Bayes) on text classification tasks (Kim, 2014; Iyyer et al., 2015; Tang et al., 2015; Dai and Le, 2015; Jin et al., 2016; Joulin et al., 2017; Shen et al., 2018). Typ- ical neural network-based text classification mod- els are based on words. They typically use words in the target documents as inputs, map words into continuous vectors (embeddings), and capture the semantics in documents by using compositional functions over word embeddings such as averag- ing or summation of word embeddings, convolu- tional neural networks (CNN), and recurrent neu- ral networks (RNN). Apart from the aforementioned approaches, past studies attempted to use entities in a knowl- edge base (KB) (e.g., Wikipedia) to capture the semantics in documents. These models typi- cally represent a document by using a set of en- tities (or bag of entities) relevant to the document (Gabrilovich and Markovitch, 2006, 2007; Xiong et al., 2016). The main benefit of using entities instead of words is that unlike words, entities pro- vide unambiguous semantic signals because they are uniquely identified in a KB. One key issue here is to determine the way in which to associate a document with its relevant entities. An exist- ing straightforward approach (Peng et al., 2016; Xiong et al., 2016) involves creating a set of rele- vant entities using an entity linking system to de- tect and disambiguate the names of entities in a document. However, this approach is problematic because (1) entity linking systems produce disam- biguation errors (Cornolti et al., 2013), and (2) en- tities appearing in a document are not necessar- …

Content truncated to 500 words