Making Neural QA as Simple as Possible but not Simpler

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271–280, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Making Neural QA as Simple as Possible but not Simpler Dirk Weissenborn Georg Wiese Language Technology Lab, DFKI Alt-Moabit 91c Berlin, Germany {dirk.weissenborn, georg.wiese, laura.seiffe}@dfki.de Laura Seiffe Abstract Recent development of large-scale ques- tion answering (QA) datasets triggered a substantial amount of research into end-to- end neural architectures for QA. Increas- ingly complex systems have been con- ceived without comparison to simpler neu- ral baseline systems that would justify their complexity. In this work, we propose a simple heuristic that guides the develop- ment of neural baseline systems for the ex- tractive QA task. We find that there are two ingredients necessary for building a high-performing neural QA system: first, the awareness of question words while processing the context and second, a com- position function that goes beyond simple bag-of-words modeling, such as recurrent neural networks. Our results show that FastQA, a system that meets these two re- quirements, can achieve very competitive performance compared with existing mod- els. We argue that this surprising finding puts results of previous systems and the complexity of recent QA datasets into per- spective. 1 Introduction Question answering is an important end-user task at the intersection of natural language processing (NLP) and information retrieval (IR). QA systems can bridge the gap between IR-based search en- gines and sophisticated intelligent assistants that enable a more directed information retrieval pro- cess. Such systems aim at finding precisely the piece of information sought by the user instead of documents or snippets containing the answer. A special form of QA, namely extractive QA, deals with the extraction of a direct answer to a question from a given textual context. The creation of large-scale, extractive QA datasets (Rajpurkar et al., 2016; Trischler et al., 2017; Nguyen et al., 2016) sparked research in- terest into the development of end-to-end neural QA systems. A typical neural architecture consists of an embedding-, encoding-, interaction- and an- swer layer (Wang and Jiang, 2017; Yu et al., 2017; Xiong et al., 2017; Seo et al., 2017; Yang et al., 2017; Wang et al., 2017). Most such systems de- scribe several innovations for the different layers of the architecture with a special focus on devel- oping powerful interaction layer that aims at mod- eling word-by-word interaction between question and context. Although a variety of extractive QA systems have been proposed, there is no competitive neu- ral baseline. Most systems were built in what we call a top-down process that proposes a complex architecture and validates design decisions by an ablation study. Most ablation studies, however, re- move only a single part of an overall complex ar- chitecture and therefore lack comparison to a rea- sonable neural baseline. This gap raises the ques- tion whether the complexity of current systems is justified solely by their empirical results. Another important observation is the fact that seemingly complex questions might be …

Content truncated to 500 words