Do Question Answering Modeling Improvements Hold Across Benchmarks?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13186–13218 July 9-14, 2023 ©2023 Association for Computational Linguistics Do Question Answering Modeling Improvements Hold Across Benchmarks? Nelson F. Liu♠ Tony Lee♠ Robin Jia♥ Percy Liang♠ ♠Computer Science Department, Stanford University, Stanford, CA ♥Department of Computer Science, University of Southern California, Los Angeles, CA {nfliu, tonyhlee, pliang}@cs.stanford.edu robinjia@usc.edu Abstract Do question answering (QA) modeling im- provements (e.g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks? To study this question, we introduce the notion of concurrence—two benchmarks have high con- currence on a set of modeling approaches if they rank the modeling approaches similarly. We measure the concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches and find that human-constructed benchmarks have high concurrence amongst themselves, even if their passage and ques- tion distributions are very different. Surpris- ingly, even downsampled human-constructed benchmarks (i.e., collecting less data) and programmatically-generated benchmarks (e.g., cloze-formatted examples) have high concur- rence with human-constructed benchmarks. These results indicate that, despite years of in- tense community focus on a small number of benchmarks, the modeling improvements stud- ied hold broadly. 1 Introduction The NLP community has created a diverse land- scape of extractive question answering (QA) benchmarks—their context passages may come from different sources, their questions may focus on different phenomena or be written by different populations, or other aspects of the data collection process may differ. Driven to improve benchmark performance, researchers have proposed a variety of QA modeling approaches. However, not all benchmarks receive equal attention from the com- munity (Koch et al., 2021); many QA modeling approaches are developed on a small handful of benchmarks, especially those with popular leader- boards (e.g., SQuAD; Rajpurkar et al., 2016). As a result, it is conceivable that some modeling im- provements may not hold because they are (perhaps inadvertently) benchmark-specific, while others 70 80 90 70 75 80 85 90 CBT (NE) r = 0.97 = 0.88 r = 0.97 = 0.88 70 80 90 60 70 80 EM Downsampled SQuAD (20K) r = 1.00 = 0.94 r = 1.00 = 0.94 70 80 90 60 65 70 MRQA NaturalQuestions r = 0.98 = 0.84 r = 0.98 = 0.84 70 80 90 45 50 55 60 EM MRQA NewsQA r = 0.98 = 0.87 r = 0.98 = 0.87 SQuAD 1.1 EM non-pretrained models pre-trained models Figure 1: Two benchmarks have high concurrence if they rank a set of modeling approaches similarly. Surprisingly, we find that human-constructed bench- marks (e.g., SQuAD, NaturalQuestions) have high con- currence with other human-constructed benchmarks, downsampled human-constructed benchmarks, and even programmatically-generated cloze benchmarks (e.g., the Children’s Book Test; CBT). In addition, we are able to construct synthetic benchmarks that have high concurrence with human-constructed benchmarks despite lacking natural language passages or questions. (e.g., pre-training on more data) hold more broadly. In this work, we evaluate whether improvements from modeling approaches hold (e.g., choices …

Content truncated to 500 words