Publishing the Trove Newspaper Corpus

Publishing the Trove Newspaper Corpus Steve Cassidy Department of Computing Macquarie University Sydney, Australia Abstract The Trove Newspaper Corpus is derived from the National Library of Australia’s digital archive of newspaper text. The corpus is a snapshot of the NLA collection taken in 2015 to be made available for language research as part of the Alveo Virtual Laboratory and contains 143 million articles dating from 1806 to 2007. This paper describes the work we have done to make this large corpus available as a research collection, facilitating access to individual documents and enabling large scale processing of the newspaper text in a cloud-based environment. Keywords: newspaper, corpus, linked data 1. Introduction This paper describes a new corpus of Australian histori- cal newspaper text and the process by which it was pre- pared for publication as an online resource for language research. The corpus itself is of interest as it represents a large collection of newspaper text dating from 1806 to around 2007. However, to make this data available in a us- able form, rather than a very large download, we have taken steps to build a usable web-based interface to the individ- ual documents and to facilitate large scale processing of the text in a cloud-based environment. 2. The Trove Corpus Trove1 is the digital document archive of the National Li- brary of Australia (Holley, 2010) and contains a variety of document types such as books, journals and newspapers. The newspaper archive in Trove consists of scanned ver- sions of each page as PDF documents along with a tran- scription generated by ABBYY FineReader2, which is is a state-of-the-art commercial optical character recognition (OCR) system. OCR is inherently error-prone and the qual- ity of the transcriptions varies a lot across the archive; in particular, the older samples are of poorer quality due to the degraded nature of the original documents. To help improve the quality of the OCR transcriptions, Trove provides a web based interface to allow members of the public to correct the transcriptions. This crowdsourcing approach produces a large number of corrections to news- paper texts and the quality of the collection is constantly improving. As of this writing, the Trove website reports a total of 170 million corrections to newspaper texts3. As part of this project, a snapshot sample of the Trove newspaper archive has been provided to be ingested into the Alveo Virtual Laboratory (Cassidy et al., 2014) for use in language research. One motivation for this is to provide a snapshot archive of Trove that can be used in academic research; this collection won’t change and so can be used 1http://trove.nla.gov.au/ 2http://www.abbyy.com 3http://trove.nla.gov.au/system/stats? env=prod&redirectGroupingType=island#links to reproduce published results. Alveo also aims to provide access to the data in a way that facilitates automatic pro- cessing of the text rather than the document-by-document interface provided by the Trove web API. The snapshot we were given of the current state of the col- lection contains around 143 million articles from 836 dif- ferent newspaper titles. The collection …

Content truncated to 500 words