Plug-and-Play Document Modules for Pre-trained Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15713–15729 July 9-14, 2023 ©2023 Association for Computational Linguistics Plug-and-Play Document Modules for Pre-trained Models Chaojun Xiao1,2,3, Zhengyan Zhang1,2,3, Xu Han1,2,3∗, Chi-Min Chan1,2,3, Yankai Lin4,5 Zhiyuan Liu1,2,3∗, Xiangyang Li6, Zhonghua Li6, Zhao Cao6, Maosong Sun1,2,3∗ 1NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing 2International Innovation Center of Tsinghua University, Shanghai 3Quan Cheng Laboratory 4Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 5Beijing Key Laboratory of Big Data Management and Analysis Methods 6Huawei Technologies Co., Ltd. xiaocj20@mails.tsinghua.edu.cn, {hanxu2022,liuzy,sms}@tsinghua.edu.cn Abstract Large-scale pre-trained models (PTMs) have been widely used in document-oriented NLP tasks, such as question answering. However, the encoding-task coupling requirement results in the repeated encoding of the same documents for different tasks and queries, which is highly computationally inefficient. To this end, we tar- get to decouple document encoding from down- stream tasks, and propose to represent each document as a plug-and-play document mod- ule, i.e., a document plugin, for PTMs (PlugD). By inserting document plugins into the back- bone PTM for downstream tasks, we can en- code a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simulta- neously encode documents and input queries using task-specific encoders. Extensive exper- iments on 8 datasets of 4 typical NLP tasks show that PlugD enables models to encode doc- uments once and for all across different sce- narios. Especially, PlugD can save 69% com- putational costs while achieving comparable performance to state-of-the-art encoding-task coupling methods. Additionally, we show that PlugD can serve as an effective post-processing way to inject knowledge into task-specific mod- els, improving model performance without any additional model training. Our code and check- points can be found in https://github.com/ thunlp/Document-Plugin. 1 Introduction In recent years, large-scale pre-trained models (PTMs) (Devlin et al., 2019; Raffel et al., 2020) have been widely adopted and achieved break- through performance for document-oriented NLP tasks, such as question answering. However, due to the tight coupling of document encoding and concrete tasks, PTMs have to dynamically gener- ate document representations according to specific tasks and queries, leading to the repeated encoding ∗Corresponding authors. Where did Shakespeare born? Warwickshire Document Plugin PTM Pre-trained Model PTM Slot Filling Question Answering Document Encoding QA <Shakespeare, spouse, ___ > Anne Hathaway SF Fact Verification QA Task-specific Models Shakespeare was an playwriter. FV Yes Figure 1: Illustration of plug-and-play document mod- ules. Document encoding is decoupled from concrete tasks. By plugging document plugins into task-specific models, we can handle multiple tasks such as question answering, fact verification, and slot filling. of the same documents in different applications. For example, Wikipedia documents are commonly used in various knowledge-intensive tasks such as question answering (Chen et al., 2017), fact verifi- cation (Thorne et al., 2018), and dialogue genera- tion (Dinan et al., 2019). In this case, existing meth- ods separately encode one document for each task or even for each input query (e.g., a question for …

Content truncated to 500 words