How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN R. Thomas McCoy,∗1 Paul Smolensky,2,3 Tal Linzen,4 Jianfeng Gao,2 Asli Celikyilmaz†5 1Princeton University, USA 2Microsoft Research, USA 3Johns Hopkins University, USA 4New York University, USA 5Meta AI, USA tom.mccoy@princeton.edu, psmo@microsoft.com, linzen@nyu.edu, jfgao@microsoft.com, aslic@meta.com Abstract Current language models can generate high- quality text. Are they simply copying text they have seen before, or have they learned gener- alizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential struc- ture (n-grams) and syntactic structure. We ap- ply these analyses to four neural language models trained on English (an LSTM, a Trans- former, Transformer-XL, and GPT-2). For local structure—e.g., individual dependencies— text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model’s test set. For larger-scale structure— e.g., overall sentence structure—model- generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also per- form extensive manual analysis, finding evi- dence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2’s novel text is usually well-formed morphologically and syntacti- cally but has reasonably frequent semantic issues (e.g., being self-contradictory). 1 Introduction There are many abstract properties that charac- terize well-formed text, from grammatical prop- ∗ Work done while at Microsoft Research and Johns Hopkins University. † Work done while at Microsoft Research. erties (e.g., subject-verb agreement) to discourse properties (e.g., coherence). How can we tell which of these properties have been learned by a language model (LM)? One popular approach is to analyze text generated by the LM (Dai et al., 2019; Brown et al., 2020; Zhang et al., 2022). The assumption underlying this approach is that, if the text displays a particular linguistic property (e.g., coherence), then the LM must have cap- tured that property. We argue that this approach has an important shortcoming: The generated text could have been copied from the LM’s training data, in which case it does not provide clear evidence for lin- guistic abstractions. For example, suppose an LM achieves coherence by copying a paragraph from its training set. In this case, the entity that deserves credit for being coherent would not be the LM but rather the human who originally wrote that paragraph. To address this concern, it is important to check whether LM-generated text duplicates from the training data. That is, we argue that LM- generated text must have two traits to be clear evidence that the LM has learned some abstrac- tion A: (1) Quality: The text must be well-formed with respect to A. (2) Novelty: The text must not have been copied from the training data. Much prior work has discussed how to evaluate various aspects of quality (Gatt and Krahmer, 2018; Celikyilmaz et al., 2020). …

Content truncated to 500 words