Evaluation Paradigms in Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9630–9642 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9630 Evaluation Paradigms in Question Answering Pedro Rodriguez University of Maryland Facebook Reality Labs me@pedro.ai Jordan Boyd-Graber CS, LSC, iSchool, UMIACS University of Maryland jbg@umiacs.umd.edu Abstract Question answering (QA) primarily descends from two branches of research: (1) Alan Tur- ing’s investigation of machine intelligence at Manchester University and (2) Cyril Clever- don’s comparison of library card catalog in- dices at Cranfield University. This position paper names and distinguishes these paradigms. Despite substantial overlap, subtle but signifi- cant distinctions exert an outsize influence on research. While one evaluation paradigm val- ues creating more intelligent QA systems, the other paradigm values building QA systems that appeal to users. By better understand- ing the epistemic heritage of QA, researchers, academia, and industry can more effectively accelerate QA research. 1 Introduction This position paper seeks to answer the question why do we do question answering and understand the consequences of different answers to this ques- tion. Our primary contribution is to outline two dis- tinct and common reasons that motivate researchers to pursue question answering (QA)—the Cran- field and Manchester paradigms. The Cranfield paradigm is not new (Section 2): it has a long and storied history in information retrieval (Voorhees, 2019). Here, we describe why a large share of QA is implicitly motivated by serving the needs of users, which is exactly the Cranfield paradigm (although most do not say so explicitly). Section 3 christens another paradigm—the Manchester paradigm—at home in the more eclec- tic corners of QA: to test and inculcate intelligence. These paradigms have much in common (Sec- tion 4), which helps explains why this distinction is not immediately apparent. However, the differ- ences (Section 5) are ignored at your own peril. Section 6 articulates how the community can better heed the distinction and how the paradigms can inform each other. 2 Serving Users: The Cranfield Paradigm Let’s start with the Cranfield paradigm, named after Cranfield University in Bedfordshire. The Cran- field “experimental tradition founded by a librar- ian, working with card indexes, a half-century ago” spurred a revolution in information retrieval evalu- ation (Robertson, 2008). In information retrieval, a “system locates infor- mation that is relevant to a user’s query” (Sander- son and Croft, 2012). The most natural IR evalua- tion is to ask users whether documents satisfy their information need. However, much like annotation in NLP, this is expensive and time-consuming, so Cleverdon (1967) proposes an alternative. Rather than have users interact with every potential sys- tem, build re-usable test collections and evaluate all systems by re-using the same collection. Al- though “obvious” to twenty-first century readers, the use of offline test collections for evaluation was controversial (Taube, 1965), and the approach is still debated (Saracevic, 2007). Rather than putting users in front of every IR system, in the Cranfield paradigm, researchers perform ex- periments on test collections to compare the rela- tive effectiveness of different retrieval approaches …

Content truncated to 500 words