Learning Label Modular Prompts for Text Classification in the Wild

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1677 - 1690 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning Label Modular Prompts for Text Classification in the Wild Hailin Chen♣♠, Amrita Saha♠, Shafiq Joty♣♠, Steven C.H. HOI♠ ♣ Nanyang Technological University, Singapore ♠ Salesforce Research {hailin001, srjoty}@ntu.edu.sg {amrita.saha, shoi}@salesforce.com Abstract Machine learning models usually assume i.i.d data during training and testing, but data and tasks in real world often change over time. To emulate the transient nature of real world, we propose a challenging but practical task: text classification in-the-wild, which introduces different non-stationary training/testing stages. Decomposing a complex task into modular components can enable robust generalisation under such non-stationary environment. How- ever, current modular approaches in NLP do not take advantage of recent advances in pa- rameter efficient tuning of pretrained language models. To close this gap, we propose MOD- ULARPROMPT, a label-modular prompt tuning framework for text classification tasks. In MOD- ULARPROMPT, the input prompt consists of a sequence of soft label prompts, each encod- ing modular knowledge related to the corre- sponding class label. In two of most formidable settings, MODULARPROMPT outperforms rele- vant baselines by a large margin demonstrating strong generalisation ability. We also conduct comprehensive analysis to validate whether the learned prompts satisfy properties of a modular representation.1 1 Introduction While NLP research has received a significant boost in performance by employing large-scale pre- trained language models (PLMs), finetuning an entire dedicated model for each task is not always practical or even feasible, especially as model size continues to grow. To alleviate this, recently there has been increased interest in parameter-efficient methods such as Adapter (Houlsby et al., 2019) and Prompt-based tuning methods (Lester et al., 2021; Qin and Eisner, 2021; Liu et al., 2021; Li and Liang, 2021). When training on a downstream task, these methods keep the PLM frozen and only 1Our Code is available at https://github.com/ salesforce/ModularPrompt Figure 1: Example of MODULARPROMPT for stage-fused NER. Top 2 blocks: training stages covering disjoint sets of entity types (e.g., event, person). Bottom block: fused test stage covering entity types of person and organization from two training stages. Coloured boxes denote label prompts. Underlining and italic blue denotes named entity and its type. update a small set of parameters that are added to the network. Among these methods, PROMPTTUN- ING (Lester et al., 2021) proposes tunable prompts – a sequence of learnable soft tokens, and show impressive results, even competitive to full-model finetuning with a large PLM. While these prompt-based methods have proven quite effective on popular benchmarks, these stan- dard tasks typically assume only independently and identically distributed (i.i.d) data during training and testing. However, practical cognitive tasks in the real world are usually more complex involving changing contexts or non-stationary environments. Talking about what is next for NLP, Kathleen McK- eown in a recent interview (source) said: “Most models are static. But the world changes every minute, every second. Dealing with a dynamic world is a new …

Content truncated to 500 words