CCOHA: Clean Corpus of Historical American English

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6958–6966 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6958 CCOHA: Clean Corpus of Historical American English Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, Sabine Schulte im Walde Institute for Natural Language Processing University of Stuttgart Pfaffenwaldring 5b, 70569 Stuttgart, Germany {alatrarm, schlecdk, jonas, schulte}@ims.uni-stuttgart.de Abstract Modelling language change is an increasingly important area of interest within the fields of sociolinguistics and historical linguistics. In recent years, there has been a growing number of publications whose main concern is studying changes that have occurred within the past centuries. The Corpus of Historical American English (COHA) is one of the most commonly used large corpora in diachronic studies in English. This paper describes methods applied to the downloadable version of the COHA corpus in order to overcome its main limitations, such as inconsistent lemmas and malformed tokens, without compromising its qualitative and distributional properties. The resulting clean corpus of historical American English (CCOHA) contains a larger number of cleaned word tokens which can offer better insights into language change and allow for a larger variety of tasks to be performed. Keywords: COHA, Corpora, Historical Linguistics, Language Change 1. Introduction Languages are in a constant process of evolution. That is, they constantly change over time on all levels of linguis- tic structure. These changes reflect—and are driven by— external factors such as cultural changes and technological advances (Blank, 1999; Fromkin et al., 2018). The field of historical or diachronic linguistics is concerned with the study and analysis of language change over time. Over the past two decades, researchers have shown an increased in- terest in the various aspects of diachronic language change. This can be attributed to the advances in technology such as the digitization of historical texts, improved computational power and availability of large-scale historical corpora de- signed specifically for diachronic studies (Tahmasebi et al., 2018; Tang, 2018; Bowern, 2019). Large historical cor- pora first appeared a decade ago and quickly gained pop- ularity because they allow researchers to test hypotheses using computational approaches that are only possible with corpora of such volume (Kutuzov et al., 2018; Dubossarsky et al., 2019; Perrone et al., 2019; Schlechtweg et al., 2019). The Corpus of Historical American English (COHA) (Davies, 2012) is a popular large-scale resource for study- ing lexical, syntactic and semantic change in English. De- spite its many features and advantages, COHA is not with- out its limitations. These shortcomings, which include in- consistent lemmas and malformed tokens, can complicate certain tasks and increase the required time and effort to complete them. As a case in point, let us consider the original task for which we needed COHA. The task re- quired sentence-level context extraction for a set of target words, but was hindered by the presence of malformed to- kens around sentence boundaries. To clarify, let us con- sider Example 1. which shows two sentences that have been merged due to the boundary loss …

Content truncated to 500 words