Online Near-Duplicate Detection of News Articles

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1242–1249 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1242 Online Near-Duplicate Detection of News Articles Simon Rodier and Dave Carter Digital Technologies, National Research Council Canada {simon.rodier, david.carter}@cnrc-nrc.gc.ca Abstract Near-duplicate documents are particularly common in news media corpora. Editors often update wirefeed articles to address space constraints in print editions or to add local context; journalists often lightly modify previous articles with new information or minor corrections. Near-duplicate documents have potentially significant costs, including bloating corpora with redundant information (biasing techniques built upon such corpora) and requiring additional human and computational analytic resources for marginal benefit. Filtering near-duplicates out of a collection is thus important, and is particularly challenging in applications that require them to be filtered out in real-time with high precision. Previous near-duplicate detection methods typically work offline to identify all near-duplicate pairs in a set of documents. We propose an online system which flags a near-duplicate document by finding its most likely original. This system adapts the shingling algorithm proposed by Broder (1997), and we test it on a challenging dataset of web-based news articles. Our online system presents state-of-the-art F1-scores, and can be tuned to trade precision for recall and vice-versa. Given its performance and online nature, our method can be used in many real-world applications. We present one such application, filtering near-duplicates to improve productivity of human analysts in a situational awareness tool. Keywords: duplicate detection, near-duplicate detection, news corpora, natural language processing, situational awareness 1. Introduction Near-duplicates are multi-sentence spans of text in the same language that convey the same content using largely similar vocabulary; parts may be identical, paraphrased or lightly modified, and limited portions may be entirely unique to one document but not its near-duplicates. Near-duplicates can be thought of as documents with a non-trivial but bounded edit distance. This paper presents a near-duplicate detection system that is online, rapid, and has been vali- dated in an enterprise software tool. Near-duplicate documents are prevalent in some corpora and come at a significant cost. There are computational costs in processing documents that provide no new infor- mation; and significant costs to end users, who are forced to sift through redundant documents when trying to accom- plish a task. Machine learning algorithms may also be sus- ceptible to error due to near-duplicate documents. For ex- ample, Chatterjee (2019) describes how a sudden spike in news about Brexit has thrown off machine-based trading systems; a near-duplicate detection system could be used to filter out fundamentally similar documents to help these systems cope with bursts of news articles. We build upon the shingling method proposed by Broder (1997) to create a real-time near-duplicate detection system that offers state-of-the-art results, and which can be tuned to provide better precision or recall as needed. 1.1. Near-duplicate detection There are different definitions of near-duplicate documents. For web-based news documents, Theobald et al. (2008) de- scribe one type as consisting of …

Content truncated to 500 words