Task-guided Disentangled Tuning for Pretrained Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 3126 - 3137 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Task-guided Disentangled Tuning for Pretrained Language Models Jiali Zeng1∗, Yufan Jiang1, Shuangzhi Wu1, Yongjing Yin2, Mu Li1 1Tencent Cloud Xiaowei, Beijing, China 2Zhejiang University, Westlake University, Zhejiang, China {lemonzeng,garyyfjiang,frostwu,ethanlli}@tencent.com yinyongjing@westlake.edu.cn Abstract Pretrained language models (PLMs) trained on large-scale unlabeled corpus are typically fine- tuned on task-specific downstream datasets, which have produced state-of-the-art results on various NLP tasks. However, the data dis- crepancy issue in domain and scale makes fine- tuning fail to efficiently capture task-specific patterns, especially in the low data regime. To address this issue, we propose Task-guided Disentangled Tuning (TDT) for PLMs, which enhances the generalization of representations by disentangling task-relevant signals from the entangled representations. For a given task, we introduce a learnable confidence model to detect indicative guidance from context, and further propose a disentangled regularization to mitigate the over-reliance problem. Experimen- tal results on GLUE and CLUE benchmarks show that TDT gives consistently better results than fine-tuning with different PLMs, and ex- tensive analysis demonstrates the effectiveness and robustness of our method. Code is avail- able at https://github.com/lemon0830/TDT. 1 Introduction In recent years, pretrained language models (PLMs) trained in a self-supervised manner like mask lan- guage modeling have achieved promising results on various natural language processing (NLP) tasks (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019b), which learn general linguistic and seman- tic knowledge from massive general corpus. To adapt PLMs to specific NLP tasks, a commonly- used approach is fine-tuning, where the whole or part of model parameters are tuned by task-specific objectives. Despite its success, the fine-tuned models have been proven ineffective to capture task-specific patterns due to the gap between task- agnostic pre-training and the weak fine-tuning with limited task-specific data (Gu et al., 2020; Gururan- gan et al., 2020; Kang et al., 2020). ∗Corresponding author. Train Jobs founded apple in 1976 apple launches new apple phones apple is interested in news content and started recruiting editors on a large scale last year Label Sentence tech apple tech Test The total output of apples in arid regions has fallen, and the price of high-quality apples has risen. Pred/Label Sentence tech/finance Apple's founder's daughter bought a new mansion in San Francisco tech/house tech tech Ours Label Sentence Jobs founded apple in 1976 Jobs founded apple in 1976 Jobs founded apple in 1976 Positive: Negative: tech Figure 1: An over-reliance example of news classifica- tion task. The fine-tuned models tend to learn a simple rule that “Apple” (red) indicates “tech” class while ig- nore the real meaning of “apples” (green) A reliable model is expected to find out truly task-specific patterns (underlined words) instead of some high frequency but insignificant words (“apple”). To address this problem, most existing methods focus on adapting PLMs to downstream tasks by continual pre-training on in-domain unsupervised data (Gururangan et al., 2020; Gu et al., 2020; Wu et al., 2021; Kang et al., 2020; …

Content truncated to 500 words