Character-Level Question Answering with Attention

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1598–1607, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Character-Level Question Answering with Attention David Golub University of Washington golubd@cs.washington.edu Xiaodong He Microsoft Research xiaohe@microsoft.com Abstract We show that a character-level encoder- decoder framework can be successfully ap- plied to question answering with a structured knowledge base. We use our model for single- relation question answering and demonstrate the effectiveness of our approach on the Sim- pleQuestions dataset (Bordes et al., 2015), where we improve state-of-the-art accuracy from 63.9% to 70.9%, without use of ensem- bles. Importantly, our character-level model has 16x fewer parameters than an equivalent word-level model, can be learned with signif- icantly less data compared to previous work, which relies on data augmentation, and is ro- bust to new entities in testing. 1 1 Introduction Single-relation factoid questions are the most com- mon form of questions found in search query logs and community question answering websites (Yih et al., 2014; Fader et al., 2013). A knowledge-base (KB) such as Freebase, DBpedia, or Wikidata can help answer such questions after users reformulate them as queries. For instance, the question “Where was Barack Obama born?” can be answered by is- suing the following KB query: λ(x).place of birth(Barack Obama, x) However, automatically mapping a natural language question such as “Where was Barack Obama born?” 1Our code is publicly available at https://github. com/davidgolub/simpleqa to its corresponding KB query remains a challeng- ing task. There are three key issues that make learning this mapping non-trivial. First, there are many para- phrases of the same question. Second, many of the KB entries are unseen during training time; however, we still need to correctly predict them at test time. Third, a KB such as Freebase typically contains mil- lions of entities and thousands of predicates, mak- ing it difficult for a system to predict these entities at scale (Yih et al., 2014; Fader et al., 2014; Bordes et al., 2015). In this paper, we address all three of these issues with a character-level encoder-decoder frame- work that significantly improves performance over state-of-the-art word-level neural models, while also providing a much more compact model that can be learned from less data. First, we use a long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) encoder to em- bed the question. Second, to make our model ro- bust to unseen KB entries, we extract embeddings for questions, predicates and entities purely from their character-level representations. Character- level modeling has been previously shown to gen- eralize well to new words not seen during training (Ljubeˇsi´c et al., 2014; Chung et al., 2016), which makes it ideal for this task. Third, to scale our model to handle the millions of entities and thousands of predicates in the KB, instead of using a large out- put layer in the decoder to directly predict the entity and predicate, we use a general interaction function between the question embeddings and KB embed- dings that measures their …

Content truncated to 500 words