Deduplicating Training Data Makes Language Models Better

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8424 - 8445 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Deduplicating Training Data Makes Language Models Better Katherine Lee∗† Daphne Ippolito∗†‡ Andrew Nystrom† Chiyuan Zhang† Douglas Eck† Chris Callison-Burch‡ Nicholas Carlini† Abstract We find that existing language modeling datasets contain many near-duplicate exam- ples and long repetitive substrings. As a result, over 1% of the unprompted out- put of language models trained on these datasets is copied verbatim from the train- ing data. We develop two tools that allow us to deduplicate training datasets—for exam- ple removing from C4 a single 61 word En- glish sentence that is repeated over 60,000 times. Deduplication allows us to train mod- els that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy. We can also reduce train-test overlap, which af- fects over 4% of the validation set of stan- dard datasets, thus allowing for more accurate evaluation. Code for deduplication is released at https://github.com/google-research/ deduplicate-text-datasets. 1 Introduction A key factor behind the recent progress in natural language processing is the development of large- scale text corpora used to train increasingly large language models. These datasets have grown from single gigabytes to as much as a terabyte over the past few years (Chelba et al., 2013; Xue et al., 2020; Graff et al., 2003; Brown et al., 2020). Because it is so expensive to perform manual review and curation on massive datasets, they tend to suffer in quality compared to their smaller predecessors. This has implications far beyond metrics like per- plexity and validation loss, as learned models re- flect the biases present in their training data (Ben- der et al., 2021; Wallace et al., 2019; Sheng et al., 2020). Quantitatively and qualitatively understand- ing these datasets is therefore a research challenge in its own right (Dodge et al., 2021a). ∗ Equal contribution. † Google Research, Brain Team. ‡ University of Pennsylvania. Correspond to kather- inelee@google.com and daphnei@seas.upenn.edu. We show that one particular source of bias, du- plicated training examples, is pervasive: all four common NLP datasets we studied contained dupli- cates. Additionally, all four corresponding valida- tion sets contained text duplicated in the training set. While naive deduplication is straightforward (and the datasets we consider already perform some naive form of deduplication), performing thorough deduplication at scale is both computationally chal- lenging and requires sophisticated techniques. We propose two scalable techniques to detect and remove duplicated training data. Exact sub- string matching identifies verbatim strings that are repeated. This allows us to identify cases where only part of a training example is duplicated (§4.1). Approximate full document matching uses hash- based techniques (Broder, 1997) to identify pairs of documents with high n-gram overlap (§4.2). We identify four distinct advantages to training on datasets that have been thoroughly deduplicated. 1. Over 1% of tokens emitted unprompted from a model trained on standard datasets (e.g., C4) are …

Content truncated to 500 words