Quantifying the Effects of Text Duplication on Semantic Models

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2737–2747 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Quantifying the Effects of Text Duplication on Semantic Models Alexandra Schofield1 Laure Thompson1 David Mimno2 1 Department of Computer Science, Cornell University, Ithaca, NY {xanda, laurejt}@cs.cornell.edu 2 Department of Information Science, Cornell University, Ithaca, NY mimno@cornell.edu Abstract Duplicate documents are a pervasive prob- lem in text datasets and can have a strong effect on unsupervised models. Meth- ods to remove duplicate texts are typically heuristic or very expensive, so it is vital to know when and why they are needed. We measure the sensitivity of two latent semantic methods to the presence of dif- ferent levels of document repetition. By artificially creating different forms of du- plicate text we confirm several hypotheses about how repeated text impacts models. While a small amount of duplication is tol- erable, substantial over-representation of subsets of the text may overwhelm mean- ingful topical patterns. 1 Introduction Different discussions of the same subject tend to use similar words. Unsupervised models such as latent semantic analysis (LSA) (Deerwester et al., 1990) and latent Dirichlet allocation (LDA) (Blei et al., 2003) look for these statistical signatures of topicality in the form of repeated word co- occurrences. These methods have become increas- ingly popular because they are powerful and easy to apply to large unlabeled datasets. The appar- ent ease-of-use of LSA and LDA, however, makes it easy to overlook potential problems in text cor- pora. In this work, we focus on measuring the im- pact of one such issue: duplicate text. Latent semantic methods look for patterns of repetition. But when text is repeated exactly, sta- tistical methods that look for patterns may be di- verted from more meaningful semantic groups: verbatim repetition looks, to the algorithm, more topical than actual topics. If not accounted for, re- peated text can change measures of fitness to over- value fit on repeated texts, or even “leak” held out data that is duplicated in the training data. At best, duplication may cause us to overestimate the ex- pressiveness and reliability of models. At worst, models skewed by text duplication may invalidate any conclusions drawn from them, and, by exten- sion, the method itself. Text replication is a persistent and difficult problem in natural language corpora. In social media settings, partial duplication due to quota- tion and threading is ubiquitous. Of the 20k posts in the 20 Newsgroups corpus (Lang, 1995), 1151 are exact duplicates, and 25% of the remaining to- kens are quoted text from other newsgroup mes- sages.1 In literary corpora, different versions of the same document may also conflict: text files for Hamlet may differ slightly due to publisher information, line numbers, editorial changes be- tween Shakespeare’s folios, and footnotes. Re- moving exactly identical duplicates of texts is pos- sible through direct lexicographic matching, but for lexical near-duplicates and partial textual over- lap, we may need more careful heuristics to detect duplicates, forcing researchers …

Content truncated to 500 words