Deduplication
task
ID: deduplication
Related Papers
Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings
AbstractDeduplication is the task of identifying near and exact duplicate data items in a collection. In this paper, we present a novel method for deduplication of scholarly documents. We develop …...
Deduplicating Training Data Makes Language Models Better
AbstractWe find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these …...