Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 901–910 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 901 Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings Bikash Gyawali, Lucas Anastasiou, Petr Knoth Knowledge Media Institute The Open University Milton Keynes, United Kingdom {bikash.gyawali, lucas.anastasiou, petr.knoth}@open.ac.uk Abstract Deduplication is the task of identifying near and exact duplicate data items in a collection. In this paper, we present a novel method for deduplication of scholarly documents. We develop a hybrid model which uses structural similarity (locality sensitive hashing) and meaning representation (word embeddings) of document texts to determine (near) duplicates. Our collection constitutes a subset of mul- tidisciplinary scholarly documents aggregated from research repositories. We identify several issues causing data inaccuracies in such collections and motivate the need for deduplication. In lack of existing dataset suitable for study of deduplication of scholarly documents, we create a ground truth dataset of 100K scholarly documents and conduct a series of experiments to empirically establish optimal values for the parameters of our deduplication method. Experimental evaluation shows that our method achieves a macro F1-score of 0.90. We productionise our method as a publicly accessible web API service serving deduplication of scholarly documents in real time. Keywords: Deduplication, Scholarly Documents, Locality Sensitive Hashing, Word Embeddings, Digital Repositories 1. Introduction Publishing research findings as scholarly documents (publi- cations) has always been the mainstream model for dissem- inating scientific research. To this end, authors publish their research outputs as scholarly documents and deposit them to one or more publishing platforms, repositories, of their choice. Such choices include institutional repositories, per- sonal web pages, preprint services, academic publishers’ platform and so on. Authors choosing to submit their arti- cle to multiple repositories have different motivations to do so, such as: • Different versions of author’s manuscript become suit- able for submission to different repositories. Ex- amples include, preprint repositories for submitting manuscripts that are yet to be peer reviewed, authors’ personal web pages for hosting open access versions of their publications, etc. • Institutional as well as national policies mandate au- thors submit their research outputs to their own insti- tution’s repository. • For publications with multiple authorship, each author may decide to submit it to one or more repositories of their own choice/institution. • By submitting the same research work to multiple dif- ferent repositories, a wider audience can be reached (for example, documents from open access reposi- tories are available to everyone, while the publisher might put the document behind a paywall); research can also be disseminated sooner thereby increasing ac- celerating the scholarly communication process. While authors may submit exact duplicate copies of their research output to multiple repositories, they might also in- troduce slight variations (near duplicates) while submit- ting to different repositories (Klein et al., 2016). Through- out the remainder of this text, we will use the generic term duplicates to refer to both exact and near duplicates. For example, authors …

Content truncated to 500 words