Leveraging Knowledge Bases in LSTMs for Improving Machine Reading

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1436–1446 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1132 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1436–1446 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1132 Leveraging Knowledge Bases in LSTMs for Improving Machine Reading Bishan Yang Machine Learning Department Carnegie Mellon University bishan@cs.cmu.edu Tom Mitchell Machine Learning Department Carnegie Mellon University tom.mitchell@cs.cmu.edu Abstract This paper focuses on how to take advan- tage of external knowledge bases (KBs) to improve recurrent neural networks for machine reading. Traditional methods that exploit knowledge from KBs en- code knowledge as discrete indicator fea- tures. Not only do these features gener- alize poorly, but they require task-specific feature engineering to achieve good per- formance. We propose KBLSTM, a novel neural model that leverages continuous representations of KBs to enhance the learning of recurrent neural networks for machine reading. To effectively inte- grate background knowledge with infor- mation from the currently processed text, our model employs an attention mecha- nism with a sentinel to adaptively decide whether to attend to background knowl- edge and which information from KBs is useful. Experimental results show that our model achieves accuracies that surpass the previous state-of-the-art results for both entity extraction and event extraction on the widely used ACE2005 dataset. 1 Introduction Recurrent neural networks (RNNs), a neural archi- tecture that can operate over text sequentially, have shown great success in addressing a wide range of natural language processing problems, such as parsing (Dyer et al., 2015), named entity recogni- tion (Lample et al., 2016), and semantic role label- ing (Zhou and Xu, 2015)). These neural networks are typically trained end-to-end where the input is only text or a sequence of words and a lot of back- ground knowledge is disregarded. The importance of background knowledge in natural language understanding has long been rec- ognized (Minsky, 1988; Fillmore, 1976). Earlier NLP systems mostly exploited restricted linguistic knowledge such as manually-encoded morpholog- ical and syntactic patterns. With the advanced de- velopment of knowledge base construction, large amounts of semantic knowledge become avail- able, ranging from manually annotated semantic networks like WordNet 1 to semi-automatically or automatically constructed knowledge graphs like DBPedia 2 and NELL 3. While traditional ap- proaches have exploited the use of these knowl- edge bases (KBs) in NLP tasks (Ratinov and Roth, 2009; Rahman and Ng, 2011; Nakashole and Mitchell, 2015), they require a lot of task-specific engineering to achieve good performance. One way to leverage KBs in recurrent neural networks is by augmenting the dense representa- tions of the networks with the symbolic features derived from KBs. This is not ideal as the sym- bolic features have poor generalization ability. In addition, they can be highly sparse, e.g., using WordNet synsets can easily produce millions of indicator features, leading to high computational cost. Furthermore, the usefulness of knowledge features varies across contexts, as general KBs …

Content truncated to 500 words