BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Proceedings of NAACL-HLT 2019, pages 2924–2936 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2924 BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark∗1, Kenton Lee†, Ming-Wei Chang†, Tom Kwiatkowski† Michael Collins †2, Kristina Toutanova† ∗Paul G. Allen School of CSE, University of Washington csquared@cs.uw.edu †Google AI Language {kentonl, mingweichang, tomkwiat, mjcollins, kristout}@google.com Abstract In this paper we study yes/no questions that are naturally occurring — meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid informa- tion, and require difficult entailment-like infer- ence to solve. We also explore the effective- ness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from para- phrase or extractive QA data, and that it, sur- prisingly, continues to be very beneficial even when starting from massive pre-trained lan- guage models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human anno- tators (and 62% majority-baseline), leaving a significant gap for future work. 1 Introduction Understanding what facts can be inferred to be true or false from text is an essential part of natural language understanding. In many cases, these in- ferences can go well beyond what is immediately stated in the text. For example, a simple sentence like “Hanna Huyskova won the gold medal for Be- larus in freestyle skiing.” implies that (1) Belarus is a country, (2) Hanna Huyskova is an athlete, (3) Belarus won at least one Olympic event, (4) the USA did not win the freestyle skiing event, and so on. To test a model’s ability to make these kinds of inferences, previous work in natural language in- 1Work completed while interning at Google. 2Also affiliated with Columbia University, work done at Google. Q: Has the UK been hit by a hurricane? P: The Great Storm of 1987 was a violent extratropical cyclone which caused casualties in England, France and the Channel Islands . . . A: Yes. [An example event is given.] Q: Does France have a Prime Minister and a President? P: . . . The extent to which those decisions lie with the Prime Minister or President depends upon . . . A: Yes. [Both are mentioned, so it can be inferred both exist.] Q: Have the San Jose Sharks won a Stanley Cup? P: . . . The Sharks have advanced to the Stanley Cup fi- nals once, losing to the Pittsburgh Penguins in 2016 . . . A: No. [They were in the finals once, and lost.] Figure 1: Example yes/no questions from the BoolQ dataset. Each example consists of a question (Q), an excerpt from a passage (P), and an answer (A) with an explanation added for clarity. ference (NLI) proposed the task of labeling candi- …

Content truncated to 500 words