What Question Answering can Learn from Trivia Nerds

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422–7435 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7422 What Question Answering can Learn from Trivia Nerds Jordan Boyd-Graber , † iSchool, CS, UMIACS, LSC University of Maryland jbg@umiacs.umd.edu Benjamin Börschinger† † Google Research Zürich {jbg, bboerschinger}@google.com Abstract Question answering (QA)is not just build- ing systems; this NLP subfield also creates and curates challenging question datasets that reveal the best systems. We argue that QA datasets—and QA leaderboards—closely resemble trivia tournaments: the questions agents—humans or machines—answer reveals a “winner”. However, the research commu- nity has ignored the lessons from decades of the trivia community creating vibrant, fair, and effective QA competitions. After detail- ing problems with existing QA datasets, we outline several lessons that transfer to QA re- search: removing ambiguity, identifying better QA agents, and adjudicating disputes. 1 Introduction This paper takes an unconventional analysis to an- swer “where we’ve been and where we’re going” in question answering (QA). Instead of approach- ing the question only as computer scientists, we apply the best practices of trivia tournaments to QA datasets. The QA community is obsessed with evalua- tion. Schools, companies, and newspapers hail new SOTAs and topping leaderboards, giving rise to troubling claims (Lipton and Steinhardt, 2019) that an “AI model tops humans” (Najberg, 2018) be- cause it ‘won’ some leaderboard, putting “millions of jobs at risk” (Cuthbertson, 2018). But what is a leaderboard? A leaderboard is a statistic about QA accuracy that induces a ranking over participants. Newsflash: this is the same as a trivia tourna- ment. The trivia community has been doing this for decades (Jennings, 2006); Section 2 details this overlap between the qualities of a first-class QA dataset (and its requisite leaderboard). The ex- perts running these tournaments are imperfect, but they’ve learned from their past mistakes (see Ap- pendix A for a brief historical perspective) and cre- ated a community that reliably identifies those best at question answering. Beyond the format of the competition, trivia norms ensure individual ques- tions are clear, unambiguous, and reward knowl- edge (Section 3). We are not saying that academic QA should sur- render to trivia questions or the community—far from it! The trivia community does not under- stand the real world information seeking needs of users or what questions challenge computers. How- ever, they have well-tested protocols to declare that someone is better at answering questions than an- other. This collection of tradecraft and principles can nonetheless help the QA community. Beyond these general concepts that QA can learn from, Section 4 reviews how the “gold standard” of trivia formats, Quizbowl can improve traditional QA. We then briefly discuss how research that uses fun, fair, and good trivia questions can benefit from the expertise, pedantry, and passion of the trivia community (Section 5). 2 Surprise, this is a Trivia Tournament! “My research isn’t a silly trivia tournament,” you say. That may be, but let us first tell you a little about what …

Content truncated to 500 words