Adaptive Mixed Component LDA for Low Resource Topic Modeling

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 255–269 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 255 Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Timo Schick1,2 Hinrich Sch¨utze1 1 Center for Information and Language Processing, LMU Munich, Germany 2 Sulzer GmbH, Munich, Germany schickt@cis.lmu.de inquiries@cislmu.org Abstract Some NLP tasks can be solved in a fully unsu- pervised fashion by providing a pretrained lan- guage model with “task descriptions” in natu- ral language (e.g., Radford et al., 2019). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern- Exploiting Training (PET), a semi-supervised training procedure that reformulates input ex- amples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the result- ing training set. For several tasks and lan- guages, PET outperforms supervised training and strong semi-supervised approaches in low- resource settings by a large margin.1 1 Introduction Learning from examples is the predominant ap- proach for many NLP tasks: A model is trained on a set of labeled examples from which it then generalizes to unseen data. Due to the vast number of languages, domains and tasks and the cost of annotating data, it is common in real-world uses of NLP to have only a small number of labeled exam- ples, making few-shot learning a highly important research area. Unfortunately, applying standard supervised learning to small training sets often per- forms poorly; many problems are difficult to grasp from just looking at a few examples. For instance, assume we are given the following pieces of text: • T1: This was the best pizza I’ve ever had. • T2: You can get better sushi for half the price. • T3: Pizza was average. Not worth the price. 1Our implementation is publicly available at https:// github.com/timoschick/pet. Best pizza ever! +1 ) ∈ T ( Best pizza ever! It was . PLM great : 0.8 bad : 0.2 +1: 0.8 -1: 0.2 LCE (1) (2) Just gross. ∈ D Just gross. +1: 0.1 -1: 0.9 C (3) Figure 1: PET for sentiment classification. (1) A num- ber of patterns encoding some form of task description are created to convert training examples to cloze ques- tions; for each pattern, a pretrained language model is finetuned. (2) The ensemble of trained models anno- tates unlabeled data. (3) A classifier is trained on the resulting soft-labeled dataset. Furthermore, imagine we are told that the labels of T1 and T2 are l and l′, respectively, and we are asked to infer the correct label for T3. Based only on these examples, this is impossible because plau- sible justifications can be found for both l and l′. However, if we know that the underlying task is to identify whether the text says anything about prices, …

Content truncated to 500 words