Natural Questions: A Benchmark for Question Answering Research

Natural Questions: A Benchmark for Question Answering Research Tom Kwiatkowski♣♦♠ ∗ Jennimaria Palomaki♠ Olivia Redfield♦♠ Michael Collins♣♦♠♥ Ankur Parikh♥ Chris Alberti♥ Danielle Epstein♤♦ Illia Polosukhin♤♦ Jacob Devlin♤ Kenton Lee♥ Kristina Toutanova♥ Llion Jones♤ Matthew Kelcey♤♦ Ming-Wei Chang♥ Andrew M. Dai♣♦ Jakob Uszkoreit♣ Quoc Le♣♦ Slav Petrov♣ Google Research natural-questions@google.com Abstract We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a para- graph) and a short answer (one or more en- tities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way anno- tated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature. 1 Introduction In recent years there has been dramatic progress in machine learning approaches to problems such as machine translation, speech recognition, and image recognition. One major factor in these successes has been the development of neural methods that far exceed the performance of previous approaches. A second major factor has ∗♣Project initiation; ♦Project design; ♠Data creation; ♥Model development; ♤Project support; ♥Also affiliated with Columbia University, work done at Google; ♦No longer at Google, work done at Google. been the existence of large quantities of training data for these systems. Open-domain question answering (QA) is a benchmark task in natural language understanding (NLU), which has significant utility to users, and in addition is potentially a challenge task that can drive the development of methods for NLU. Several pieces of recent work have introduced QA data sets (e.g., Rajpurkar et al., 2016; Reddy et al., 2018). However, in contrast to tasks where it is relatively easy to gather naturally occurring examples,1 the definition of a suitable QA task, and the development of a methodology for an- notation and evaluation, is challenging. Key issues include the methods and sources used to obtain questions; the methods used to annotate and col- lect answers; the methods used to measure and ensure annotation quality; and the metrics used for evaluation. For more discussion of the limitations of previous work with respect to these issues, see Section 2 of this paper. This paper introduces Natural Questions2 (NQ), a new data set for QA research, along with methods for QA system evaluation. Our goals are three-fold: 1) To provide large-scale end-to-end training data for the QA problem. 2) To provide a data set that drives research in natural language understanding. 3) To study …

Content truncated to 500 words