Efficient Long-Text Understanding with Short-Text Models

Efficient Long-Text Understanding with Short-Text Models Maor Ivgi Uri Shaham Jonathan Berant The Blavatnik School of Computer Science, Tel-Aviv University, Israel {maor.ivgi,uri.shaham,joberant}@cs.tau.ac.il Abstract Transformer-based pretrained language mod- els (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scien- tific articles, and long documents due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on cus- tom implementations that require expensive pretraining from scratch. In this work, we pro- pose SLED: SLiding-Encoder and Decoder, a simple approach for processing long se- quences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse infor- mation across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understand- ing tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. 1 Introduction Transformer-based pretrained language models (Vaswani et al., 2017; Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020b; Brown et al., 2020) have been widely successful across all ar- eas of natural language understanding. However, applying them over long texts (such as stories, scripts, or scientific articles) is prohibitive due to their quadratic complexity in the input length. To bridge this gap, recent work has developed more efficient transformer variants (Kitaev et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020a; Guo et al., 2022) and applied them over long-range language understanding tasks (Mehta et al., 2022; Shaham et al., 2022). However, most efficient transformers use specialized architectures with custom implemen- tations that are not guaranteed to scale as well as vanilla transformers (Tay et al., 2022a). Moreover, they require an expensive pretraining step and do not exploit off-the-shelf pretrained LMs that were trained for short texts. To date, their performance on long texts has not matched the success of their short-range counterparts. In this work, we present SLED: SLiding- Encoder and Decoder, a simple yet power- ful method for applying off-the-shelf pretrained encoder-decoder models on long text problems, with a linear time and space dependency. Specifi- cally (see Figure 2), we partition long documents into overlapping chunks of tokens of constant length and encode each chunk independently with an already-pretrained encoder. Then, a pretrained decoder attends to all contextualized input rep- resentations to generate the output. Our main assumption is that input tokens can be contex- tualized through their local surrounding (using a short-text LM), and any global cross-chunk rea- soning can be handled by the decoder, similar to fusion-in-decoder (FiD) (Izacard and Grave, 2021). Our approach can be readily applied to any pretrained encoder-decoder LM such as T5 (Raffel et al., 2020b) and BART (Lewis et al., 2020) (but is not applicable …

Content truncated to 500 words