Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

Proceedings of the 28th International Conference on Computational Linguistics, pages 5569–5578 Barcelona, Spain (Online), December 8-13, 2020 5569 Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification Timo Schick Helmut Schmid Hinrich Sch¨utze Center for Information and Language Processing, LMU Munich, Germany schickt@cis.lmu.de Abstract A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels. Manually defining this mapping between words and labels re- quires both domain expertise and an understanding of the language model’s abilities. To mitigate this issue, we devise an approach that automatically finds such a mapping given small amounts of training data. For a number of tasks, the mapping found by our approach performs almost as well as hand-crafted label-to-word mappings.1 1 Introduction Pretraining language models on large corpora has led to improvements on a wide range of NLP tasks (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019, inter alia), but learning to solve tasks from only a few examples remains a challenging problem. As small datasets are common for many real- world applications of NLP, solving this challenge is crucial to enable broad applicability. A promising direction for many tasks is to reformulate them (e.g., by appending an instruction such as “translate into French”) so that they can directly be solved by a pretrained language model (Radford et al., 2019; Schick and Sch¨utze, 2020a; Brown et al., 2020). The key idea of PET (Schick and Sch¨utze, 2020a), one such approach aimed at text classification, is to rephrase each input as a cloze question for which the language model’s prediction can somehow be mapped to a label; an example is illustrated in Figure 1. While PET achieves remarkable results with little or no labeled training data, manually defining the required mapping between a language model’s predictions and labels is difficult as it requires both task- specific knowledge and an understanding of the language model’s inner workings to identify words that it understands sufficiently well. In this work, we show how this mapping can be obtained automatically, removing the need for expert knowledge: We introduce PET with Automatic Labels (PETAL), a simple approach for identifying words that can serve as proxies for labels given small amounts of training data. At its core, our approach breaks the intractable problem of finding the mapping that maximizes the likelihood of the training data into several manageable subproblems. Integrating our approach into PET significantly outperforms regular supervised training and almost matches the performance of PET with a manually defined mapping. 2 Related Work Reformulating problems as language modeling tasks has been explored in fully unsupervised settings (Radford et al., 2019; Puri and Catanzaro, 2019; Davison et al., 2019), in few-shot scenarios with limited amounts of training data (Opitz, 2019; Shwartz et al., 2020; Brown et al., 2020), and even in high-resource settings (Raffel et al., 2019). The same idea …

Content truncated to 500 words