ChapterBreak: A Challenge Dataset for Long-Range Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3704 - 3714 July 10-15, 2022 ©2022 Association for Computational Linguistics CHAPTERBREAK: A Challenge Dataset for Long-Range Language Models Simeng Sun Katherine Thai Mohit Iyyer University of Massachusetts Amherst {simengsun,kbthai,miyyer}@cs.umass.edu Abstract While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capa- bilities has not yet followed. To this end, we in- troduce CHAPTERBREAK, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter bound- ary and asks it to distinguish the beginning of the ground-truth next chapter from a set of neg- ative segments sampled from the same narra- tive. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on CHAPTERBREAK show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our CHAPTERBREAK dataset to spur more principled future research into LRLMs.1 1 Introduction Research on long-range language models (LRLMs) aims to process extremely long input sequences by making the base Transformer architecture more ef- ficient (e.g., through sparse attention, recurrence, or cached memory). These modifications are commonly validated by training LRLMs on PG- 19 (Rae et al., 2020), a long-document language modeling dataset, and demonstrating small perplex- ity decreases over shorter context models (Roy et al., 2021; ?). However, recent analysis exper- iments (Sun et al., 2021; Press et al., 2021) show that modern LRLMs rely mostly on local context (i.e., the immediately preceding 1-2K tokens) and are insensitive to various perturbations applied to more distant context. 1We make our code and data public at https:// github.com/SimengSun/ChapterBreak ... Billy Pilgrim has come unstuck in time... he has no control over where he is going... he first came unstuck in time in 1944, long before his trip to Tralfamadore... [6,608 words pass] ...Right outside the window was Billy’s own Cadillac El Dorado Coupe de Ville... The date on the license plate was1967, which would make Billy Pilgrim forty-four years old... [2,930 words pass, story shifts to World War II in 1944] ...locomotives began to move east... The war would end in May. German prisons everywhere were absolutely full... Billy Pilgrim's train... did not move for two days... [251 words pass, ch. 3 ends by shifting back to 1967] ...he traveled in time to 1967 again—to the night he was kidnapped by a flying saucer from Tralfamadore. (➕ ) Billy Pilgrim could not sleep on his daughter's wedding night. He was forty-four... [ground-truth start of ch. 4] (➖ ) Billy Pilgrim says that the Universe does not look like a lot of bright little dots to the creatures from Tralfamadore... (➖ ) All the trains were slow. The coaches stunk of coal smoke and rationed tobacco and rationed booze …

Content truncated to 500 words