QuALITY: Question Answering with Long Input Texts, Yes!

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336 - 5358 July 10-15, 2022 ©2022 Association for Computational Linguistics QuALITY: Question Answering with Long Input Texts, Yes! Richard Yuanzhe Pang∗ Alicia Parrish∗ Nitish Joshi∗ Nikita Nangia Jason Phang Angelica Chen Vishakh Padmakumar Johnny Ma Jana Thompson He He Samuel R. Bowman New York University {yzpang,alicia.v.parrish}@nyu.edu Abstract To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an aver- age length of about 5,000 tokens, much longer than typical current models can process. Un- like in prior work with passages, our ques- tions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addi- tion, only half of the questions are answerable by annotators working under tight time con- straints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%). 1 Introduction Most of the best models for natural language un- derstanding are restricted to processing only a few hundred words of text at a time, preventing them from solving tasks that require a holistic under- standing of an entire passage. Moving past this limitation would open up new applications in ar- eas like news comprehension, summarization, or applied question answering. We think that new benchmark datasets will help us do this. Most ex- isting datasets (Rajpurkar et al., 2018; Fan et al., 2019; Lelkes et al., 2021) use shorter contexts that humans can read within a few minutes. While there are open-domain QA datasets that require longer contexts (Joshi et al., 2017; Zhu et al., 2021), find- ing a short excerpt that answers the questions at hand often suffices; however, long-document QA requires understanding a long context as a whole to correctly answer questions. NarrativeQA (Koˇciský et al., 2018) is the most established existing long-text benchmark for lan- guage understanding. It’s a free-text-response QA ∗ Equal contribution. Figure 1: The crowdsourcing pipeline with an example. One writer reads the passage and writes 10 questions. Each question is validated by three or five annotators who read the full article, plus five more who have only 45 seconds per question. Writers receive feedback from both validations between writing batches. If a majority of timed annotators get the question wrong, but the un- timed annotators get it right, we classify the example as HARD and give the writer a bonus. dataset built around movie scripts and books, with an average of about 63k tokens of input per ques- tion. The authors creatively use summaries of the texts as the basis for their questions to make data collection relatively efficient. This protocol leads to short answers (avg. 4.7 tokens), and few ques- tions require complex explanation-based reason- ing: >60% are what/who questions and <10% are why/reason questions. Further, the sources …

Content truncated to 500 words