Investigating Memorization of Conspiracy Theories in Text Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4718–4729 August 1–6, 2021. ©2021 Association for Computational Linguistics 4718 Investigating Memorization of Conspiracy Theories in Text Generation Note: This paper contains examples of potentially offensive conspiracy theory text. Sharon Levy, Michael Saxon, William Yang Wang University of California, Santa Barbara sharonlevy@cs.ucsb.edu, saxon@ucsb.edu, william@cs.ucsb.edu Abstract The adoption of natural language generation (NLG) models can leave individuals vulnera- ble to the generation of harmful information memorized by the models, such as conspiracy theories. While previous studies examine con- spiracy theories in the context of social media, they have not evaluated their presence in the new space of generative language models. In this work, we investigate the capability of lan- guage models to generate conspiracy theory text. Specifically, we aim to answer: can we test pretrained generative language models for the memorization and elicitation of conspiracy theories without access to the model’s training data? We highlight the difficulties of this task and discuss it in the context of memorization, generalization, and hallucination. Utilizing a new dataset consisting of conspiracy theory topics and machine-generated conspiracy the- ories helps us discover that many conspiracy theories are deeply rooted in the pretrained lan- guage models. Our experiments demonstrate a relationship between model parameters such as size and temperature and their propensity to generate conspiracy theory text. These results indicate the need for a more thorough review of NLG applications before release and an in- depth discussion of the drawbacks of memo- rization in generative language models. 1 Introduction Recent advances in natural language processing technologies have opened a new space for individ- uals to digest information. One of these rapidly developing technologies is neural natural language generation. These models, made up of millions, or even billions (Brown et al., 2020), of parame- ters, train on large-scale datasets. While attempts are made to ensure that only “safe” data is uti- lized for training these models, several studies have shown the prevalence of biases produced by these pretrained generation models (Sheng et al., 2019; Groenwold et al., 2020; Solaiman et al., 2019). Of equally alarming concern are the memorization and subsequent generation of factually incorrect data. Conspiracy theories are one particular type of this data that can be especially damaging. While it is not new for researchers to learn that a model may memorize data (Radhakrishnan et al., 2019), we argue that the growing usage of machine learning models in society warrants targeted inves- tigation to deter potential harms from problematic data. In this paper, we address the upsides and pit- falls of memorization in generative language mod- els and its relationship with conspiracy theories. We further describe the difficulty of detecting this memorization for the categories of memorization, generalization, and hallucination. Previous stud- ies investigating memorization of text generation models have done so with access to the model’s training data (Carlini et al., 2019, 2020). As mod- els are not always published with their training datasets, we set out to examine the difficult task of eliciting …

Content truncated to 500 words