WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1535–1545, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics WIKIREADING: A Novel Large-scale Language Understanding Task over Wikipedia Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey and David Berthelot Google Research {dhewlett,allac,llion,ipolosukhin,fto,hanjay,matkelcey,dberth}@google.com Abstract We present WIKIREADING, a large-scale natural language understanding task and publicly-available dataset with 18 million instances. The task is to predict textual values from the structured knowledge base Wikidata by reading the text of the cor- responding Wikipedia articles. The task contains a rich variety of challenging clas- sification and extraction sub-tasks, mak- ing it well-suited for end-to-end models such as deep neural networks (DNNs). We compare various state-of-the-art DNN- based architectures for document classifi- cation, information extraction, and ques- tion answering. We find that models sup- porting a rich answer space, such as word or character sequences, perform best. Our best-performing model, a word-level se- quence to sequence model with a mecha- nism to copy out-of-vocabulary words, ob- tains an accuracy of 71.8%. 1 Introduction A growing amount of research in natural language understanding (NLU) explores end-to-end deep neural network (DNN) architectures for tasks such as text classification (Zhang et al., 2015), rela- tion extraction (Nguyen and Grishman, 2015), and question answering (Weston et al., 2015). These models offer the potential to remove the interme- diate steps traditionally involved in processing nat- ural language data by operating on increasingly raw forms of text input, even unprocessed char- acter or byte sequences. Furthermore, while these tasks are often studied in isolation, DNNs have the potential to combine multiple forms of reasoning within a single model. Supervised training of DNNs often requires a large amount of high-quality training data. To this end, we introduce a novel prediction task and ac- companying large-scale dataset with a range of sub-tasks combining text classification and infor- mation extraction. The dataset is made publicly- available at http://goo.gl/wikireading. The task, which we call WIKIREADING, is to pre- dict textual values from the open knowledge base Wikidata (Vrandeˇci´c and Kr¨otzsch, 2014) given text from the corresponding articles on Wikipedia (Ayers et al., 2008). Example instances are shown in Table 1, illustrating the variety of subject mat- ter and sub-tasks. The dataset contains 18.58M in- stances across 884 sub-tasks, split roughly evenly between classification and extraction (see Section 2 for more details). In addition to its diversity, the WIKIREADING dataset is also at least an order of magnitude larger than related NLU datasets. Many natural lan- guage datasets for question answering (QA), such as WIKIQA (Yang et al., 2015), have only thou- sands of examples and are thus too small for train- ing end-to-end models. Hermann et al. (2015) proposed a task similar to QA, predicting entities in news summaries from the text of the original news articles, and generated a NEWS dataset with 1M instances. The bAbI dataset (Weston et al., 2015) requires multiple forms of reasoning, but is composed of synthetically generated …

Content truncated to 500 words