ReadTwice: Reading Very Large Documents with Memories

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5189–5195 June 6–11, 2021. ©2021 Association for Computational Linguistics 5189 READTWICE: Reading Very Large Documents with Memories Yury Zemlyanskiy∗ University of Southern California yury.zemlyanskiy@usc.edu Joshua Ainslie Google Research jainslie@google.com Michiel de Jong University of Southern California msdejong@usc.edu Philip Pham Google Research phillypham@google.com Ilya Eckstein Google Research ilyaeck@google.com Fei Sha† Google Research fsha@google.com Abstract Knowledge-intensive tasks such as question answering often require assimilating informa- tion from different sections of large inputs such as books or article collections. We pro- pose READTWICE1, a simple and effective technique that combines several strengths of prior approaches to model long-range depen- dencies with Transformers. The main idea is to read text in small segments, in parallel, sum- marizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of compa- rable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books. 1 Introduction Transformer-based models such as BERT are very effective in capturing long-range dependencies in text passages through the attention mecha- nism (Vaswani et al., 2017; Devlin et al., 2019). However, the amount of compute in attention de- pends quadratically on the number of tokens in an input text passage. As such, the standard BERT implementation limits input size to a fixed number (often 512) of tokens. In reality, dependencies over significantly longer ranges are common and modeling them is crucial. For instance, in a sentence like Inside the Sammath Naur, the Ring-bearer struggled to throw the Ring into the volcano, the narrative interweaves several prior storylines from a book. Comprehending this sentence therefore requires looking up previous ∗Work is done while at Google † On leave from University of Southern California (feisha@usc.edu) 1Source code and pre-trained checkpoints for READTWICE can be found at https://goo.gle/ research-readtwice. mentions of Ring-bearer and Sammath Naur, lo- cated many tokens away. Several methods have been proposed to address this challenge; see (Tay et al., 2020) for a survey and §3 for a detailed discussion. One popular strat- egy is to reduce the number of tokens attended to. Longer inputs can in fact be processed in this way – but only up to a limit of around 5,000 tokens, as used in (Ainslie et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020) – far below the context sizes required to model long documents such as books. Another strategy such as HIBERT (Zhang et al., 2019) splits inputs into smaller segments which are processed individually, then assembled into a hierarchical representation. As a downside, inter- segment context is unavailable during encoding. We propose READTWICE, a simple approach that combines the strengths of both strategies. As its name suggests, the main idea is to process the input twice: a long text input (such as a document, or even a book) is …

Content truncated to 500 words