ReadOnce Transformers: Reusable Representations of Text for Transformers

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7129–7141 August 1–6, 2021. ©2021 Association for Computational Linguistics 7129 READONCE Transformers: Reusable Representations of Text for Transformers Shih-Ting Lin∗ Ashish Sabharwal† Tushar Khot† ∗ University of Texas, Austin, U.S.A. † Allen Institute for AI, Seattle, U.S.A. j0717lin@utexas.edu, {ashishs,tushark}@allenai.org Abstract We present READONCE Transformers, an ap- proach to convert a transformer-based model into one that can build an information- capturing, task-independent, and compressed representation of text. The resulting repre- sentation is reusable across different exam- ples and tasks, thereby requiring a document shared across many examples or tasks to only be read once. This leads to faster training and evaluation of models. Additionally, we extend standard text-to-text transformer mod- els to Representation+Text-to-text models, and evaluate on multiple downstream tasks: multi- hop QA, abstractive QA, and long-document summarization. Our one-time computed repre- sentation results in a 2x-5x speedup compared to standard text-to-text models, while the com- pression also allows existing language models to handle longer documents without the need for designing new pre-trained models. 1 Introduction Transformer-based large scale language models (LMs) (Radford et al., 2018; Devlin et al., 2019) are task-independent models that are surprisingly effective when directly fine-tuned on many differ- ent end-tasks (Rajpurkar et al., 2016; Wang et al., 2019b,a). However, this approach relies heavily on using end-task supervision to learn to solve two sub-problems simultaneously: extract information1 from an input document D and solve the end-task (e.g., answer a question about D). This incentivizes LM-based models to learn to extract only task- specific—and even example-specific—information when fine-tuned on the end-task. For example, a Question Answering (QA) model may learn to only extract the answer from D given the input question. ∗The author’s work was primarily done during an intern- ship at the Allen Institute for AI. 1By “extract information”, we mean implicitly or explicitly compute some representation of the document. Text-To-Text Model Text-To-Text Model Text-To-Text Model D1 D1 D1 Document Encoder D1 ... Task-independent Representations Task-Specific Models ReadOnce Transformers Reading Comprehension Summarization Abstractive QA Repr. + Text Model Repr. + Text Model Figure 1: READONCE Transformers: Rather than learning to extract information specific to each end- task, we use transformer-based encoders to build task- independent, reusable document representations once, and feed them into various representation+text trans- former models trained for end-tasks. This strategy, while effective on many datasets, is also inefficient. First, it requires model’s pre- trained weights to be fine-tuned separately for each end-task, even though the sub-problem of gather- ing the information content of the input document D is shared across tasks. Second, each D must be re-read from scratch in the context of each ex- ample (e.g., once for each question) even when many examples share D. Not only is this computa- tional redundancy undesirable, slow inference can quickly become a bottleneck in deployed, real-time systems if models with billions of parameters must re-read D for every input query. Inspired by …

Content truncated to 500 words