Quality Assessment of the Reuters Vol. 2 Multilingual Corpus

Quality Assessment of the Reuters Vol. 2 Multilingual Corpus Robin Eriksson University of Turku era+rcv2@iki.fi Abstract We introduce a framework for quality assurance of corpora, and apply it to the Reuters Multilingual Corpus (RCV2). The results of this quality assessment of this standard newsprint corpus reveal a significant duplication problem and, to a lesser extent, a problem with corrupted articles. From the raw collection of some 487,000 articles, almost one tenth are trivial duplicates. A smaller fraction of articles appear to be cor- rupted and should be excluded for that reason. The detailed results are being made available as on-line appendices to this article. This effort also demonstrates the beginnings of a constraint-based methodological framework for quality assessment and quality assu- rance for corpora. As a first implementation of this framework, we have investigated constraints to verify sample integrity, and to dia- gnose sample duplication, entropy aberrations, and tagging inconsistencies. To help identify near-duplicates in the corpus, we have em- ployed both entropy measurements and a simple byte bigram incidence digest. Keywords: corpus, quality assurance, quality assessment, methods, metrics 1. Introduction Corpora – machine-readable collections of language ma- terials – have an important role for the development of language technologies, especially for statistical or ma- chine-learning techniques. Because corpora are created by humans, often under severe budget and timetable cons- traints, however, we often find that they contain errors. Depending on the goals and tools of the corpus researcher, even a modest amount of errors can be problematic. In this paper, we study the Reuters Volume 2 Multilingual Corpus collection (NIST, 2005) from a quality assurance perspective. The intent is, on one hand, to attempt to document the quality and error rate of this standard corpus; and on the other, to propose improvements to the corpus itself, and to the collection process for similar corpora in the future. Finally, we provide a sketch for a general-purpose quality assurance method for corpora. 1.1. Motivation The precision in state of the art text classification is approaching the level where a handful of individual classification errors in the evaluation corpus can threaten to dominate over the error rate of the filter in a technology evaluation. In other words, the result from an experiment with an ideal 100% accurate classifier would be reported as something like 99.5% because of classification errors in the corpus – thus, it is not implausible to think that a less accurate classifier could win an evaluation. The lack of a published audit of the RCV2 corpus, and our personal observations of errors in the corpus on an anecdotal, nonscientific level, drive us to want to investig- ate the validity of the corpus, and, where possible, publish corrections for researchers who wish to exclude erroneous samples from the corpus. This should help the text classi- fication and information retrieval community by establish- ing a reasonably correct, large, real-life corpus from a single domain as a good standard benchmark collection, and eventually, a proper gold-standard test collection. Furthermore, by outlining and …

Content truncated to 500 words