Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1286 Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus Jesse Dodge♣ Maarten Sap♣♥ Ana Marasovi´c♣♥ William Agnew♥ Gabriel Ilharco♥ Dirk Groeneveld♣ Margaret Mitchell♠ Matt Gardner♣ ♥Paul G. Allen School of Computer Science & Engineering, University of Washington ♠Hugging Face ♣Allen Institute for Artificial Intelligence jessed@allenai.org Abstract Large language models have led to remark- able progress on many NLP tasks, and re- searchers are turning to ever-larger text cor- pora to train them. Some of the largest corpora available are made by scraping significant por- tions of the internet, and are frequently intro- duced with only minimal documentation. In this work we provide some of the first docu- mentation for the Colossal Clean Crawled Cor- pus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a signifi- cant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from ma- chine translation systems) and evaluation ex- amples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filter- ing disproportionately removes text from and about minority individuals. Finally, we con- clude with some recommendations for how to created and document web-scale datasets from a scrape of the internet. 1 Introduction Models pretrained on unlabeled text corpora are the backbone of many modern NLP systems (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020, inter alia). This paradigm in- centivizes the use of ever larger corpora (Kaplan et al., 2020; Henighan et al., 2020), with the biggest models now training on a substantial fraction of the publicly-available internet (Raffel et al., 2020; Brown et al., 2020). Of course, as with all ma- chine learning systems, the data such models are trained on has a large impact on their behavior. For structured, task-specific NLP datasets, best prac- tices have emerged around documenting the collec- tion process, composition, intended uses, and other Provenance Machine or human authored Social biases Documentation Levels for CommonCrawl-based Datasets Medical or health data Data contamination Demographic identities Utterance Date Metadata Included data Excluded data Figure 1: We advocate for three levels of documenta- tion when creating web-crawled corpora. On the right, we include some example of types of documentation that we provide for the C4.EN dataset. characteristics (Bender and Friedman, 2018; Gebru et al., 2018; Hutchinson et al., 2021). However, given the challenges of applying these practices to massive collections of unlabeled text scraped from the web, thorough documentation is typically not done. This leaves consumers of pretrained lan- guage models in the dark about the influences of …

Content truncated to 500 words