TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1147 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1147 TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi† Eunsol Choi† Daniel S. Weld† Luke Zettlemoyer†‡ † Paul G. Allen School of Computer Science & Engineering, Univ. of Washington, Seattle, WA {mandar90, eunsol, weld, lsz}@cs.washington.edu ‡ Allen Institute for Artificial Intelligence, Seattle, WA lukez@allenai.org Abstract We present TriviaQA, a challenging reading comprehension dataset contain- ing over 650K question-answer-evidence triples. TriviaQA includes 95K question- answer pairs authored by trivia enthusi- asts and independently gathered evidence documents, six per question on average, that provide high quality distant super- vision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sen- tences, and (3) requires more cross sen- tence reasoning to find answers. We also present two baseline algorithms: a feature- based classifier and a state-of-the-art neu- ral network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that Trivi- aQA is a challenging testbed that is worth significant future study.1 1 Introduction Reading comprehension (RC) systems aim to an- swer any question that could be posed against the facts in some reference text. This goal is challeng- ing for a number of reasons: (1) the questions can be complex (e.g. have highly compositional se- mantics), (2) finding the correct answer can re- quire complex reasoning (e.g. combining facts from multiple sentences or background knowl- edge) and (3) individual facts can be difficult to 1Data and code available at http://nlp.cs. washington.edu/triviaqa/ Question: The Dodecanese Campaign of WWII that was an attempt by the Allied forces to capture islands in the Aegean Sea was the inspiration for which acclaimed 1961 commando film? Answer: The Guns of Navarone Excerpt: The Dodecanese Campaign of World War II was an attempt by Allied forces to capture the Italian- held Dodecanese islands in the Aegean Sea following the surrender of Italy in September 1943, and use them as bases against the German-controlled Balkans. The failed campaign, and in particular the Battle of Leros, inspired the 1957 novel The Guns of Navarone and the successful 1961 movie of the same name. Question: American Callan Pinckney’s eponymously named system became a best-selling (1980s-2000s) book/video franchise in what genre? Answer: Fitness Excerpt: Callan Pinckney was an American fitness pro- fessional. She achieved unprecedented success with her Callanetics exercises. Her 9 books all became inter- national best-sellers and the video series that followed went on to sell over 6 million copies. Pinckney’s first video release ”Callanetics: 10 Years Younger In 10 Hours” outsold every other fitness video …

Content truncated to 500 words