Generating Datasets with Pretrained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6943–6951 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6943 Generating Datasets with Pretrained Language Models Timo Schick and Hinrich Schütze Center for Information and Language Processing LMU Munich, Germany schickt@cis.lmu.de Abstract To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pre- training objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for la- beled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetun- ing much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.1 1 Introduction While pretrained language models (PLMs) achieve strong results for many NLP tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019), they do not produce good sentence embeddings out of the box (Reimers and Gurevych, 2019). Recent approaches address this by augmenting or replacing the language modeling objective with likewise un- supervised sentence-level objectives (e.g., Zhang et al., 2020; Li et al., 2020), but they typically lag behind their supervised counterparts trained on human-annotated sentence pairs. Unfortunately, obtaining large amounts of high-quality training data can be both difficult and prohibitively expen- sive (Bowman et al., 2015; Agirre et al., 2016). Furthermore, with larger and larger model sizes (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020; Fedus et al., 2021), it becomes increas- ingly challenging to finetune PLMs. 1Our code and datasets are publicly available at https: //github.com/timoschick/dino. Task:Write two sentences that mean the same thing. Sentence 1: “A man is playing a flute.” Sentence 2: “He’s playing a flute.” Task:Write two sentences that are somewhat similar. Sentence 1: “A man is playing a flute.” Sentence 2: “A woman has been playing the violin.” Task: Write two sentences that are on completely different topics. Sentence 1: “A man is playing a flute.” Sentence 2: “A woman is walking down the street.” Figure 1: Continuations generated by GPT2-XL with DINO for three different task descriptions. We investi- gate two different unsupervised approaches to generat- ing sentence-similarity datasets: (i) The input sentence is given and only the continuation is generated. This requires that an (unlabeled) set of sentences is available. (ii) Both input sentence and continuation are generated. This does not rely on the availability of any resources. To alleviate both problems, we explore a novel approach to obtaining high-quality sentence em- beddings: We mimic the creation of NLI datasets by human crowdworkers (Bowman et al., 2015; Williams et al., 2018), but replace human annota- tors with large PLMs. This allows us to automat- …

Content truncated to 500 words