SCROLLS: Standardized CompaRison Over Long Language Sequences

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007 - 12021 December 7-11, 2022 ©2022 Association for Computational Linguistics SCROLLS: Standardized CompaRison Over Long Language Sequences Uri Shahamτ Elad Segalτ Maor Ivgiτ Avia Efratτ Ori Yoranτ Adi Havivτ Ankit Guptaβ Wenhan Xiongµ Mor Gevaτα Jonathan Berantτ Omer Levyτµ τ The Blavatnik School of Computer Science, Tel Aviv University α Allen Institute for AI β IBM Research µ Meta AI Abstract NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a consider- able amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We exam- ine existing long-text datasets, and handpick ones where the text is naturally long, while pri- oritizing tasks that involve synthesizing infor- mation across the input. SCROLLS contains summarization, question answering, and nat- ural language inference tasks, covering multi- ple domains, including literature, science, busi- ness, and entertainment. Initial baselines, in- cluding Longformer Encoder-Decoder, indi- cate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model ar- chitecture and pretraining methods.1 1 Introduction Standard benchmarks à la GLUE (Wang et al., 2018, 2019), WMT (Barrault et al., 2019, 2020), and SQuAD (Rajpurkar et al., 2016, 2018), have driven progress in natural language processing of short utterances. However, a large portion of nat- ural language is produced in the context of longer discourses, such as books, articles, meeting tran- scripts, etc. To tackle the computational challenges associated with processing such long sequences, a plethora of new model architectures have re- cently emerged (Tay et al., 2020b; Fournier et al., 2021), without establishing a standard scheme for evaluating them on long natural language prob- lems. Some long-context models are evaluated via language modeling perplexity, but this metric mostly captures model sensitivity to local, short- range patterns (Khandelwal et al., 2018; Sun et al., 1https://www.scrolls-benchmark.com 10 1 10 2 10 3 10 4 10 5 Words per Input (Log Scale) NarrativeQA QMSum GovReport SummScreen QuALITY Qasper ContractNLI CNN/DM SQuAD MultiNLI BERT GPT3 Popular Datasets SCROLLS Figure 1: The distribution of words per input in SCROLLS datasets (blue), alongside frequently-used NLP datasets (pink). Dashed vertical lines indicate the maximal sequence length (in tokens) of BERT (Devlin et al., 2019) and GPT3 (Brown et al., 2020). 2021). Other studies rely on Long Range Arena (Tay et al., 2021), which is limited from a natural- language perspective, since only two of its datasets involve natural language, and those are artificially- elongated through byte tokenization. To enable the research community to go beyond sentences and paragraphs, we present a new benchmark, SCROLLS: Standardized CompaRison Over Long Language Sequences. SCROLLS incorporates multiple tasks (summa- rization, question answering, and natural language inference) over various domains (literature, meet- ing transcripts, TV shows, scientific articles, and more), where each example’s input typically con- tains thousands …

Content truncated to 500 words