AMALGUM – A Free, Balanced, Multilayer English Web Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5267–5275 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5267 AMALGUM – A Free, Balanced, Multilayer English Web Corpus Luke Gessler, Siyao Peng, Yang Liu, Yilun Zhu, Shabnam Behzad, Amir Zeldes Corpling Lab Georgetown University {lg876, sp1184, yl879, yz565, sb1796, az364}@georgetown.edu Abstract We present a freely available, genre-balanced English web corpus totaling 4M tokens and featuring a large number of high-quality automatic annotation layers, including dependency trees, non-named entity annotations, coreference resolution, and discourse trees in Rhetorical Structure Theory. By tapping open online data sources the corpus is meant to offer a more sizable alternative to smaller manually created annotated data sets, while avoiding pitfalls such as imbalanced or unknown composition, licensing problems, and low-quality natural language processing. We harness knowledge from multiple annotation layers in order to achieve a “better than NLP” benchmark and evaluate the accuracy of the resulting resource. Keywords: web corpora, annotation, discourse, coreference, NER, RST 1. Introduction Corpus development for academic research and practical NLP applications often meets with a dilemma: develop- ment of high quality data sets with rich annotation schemes (e.g. MASC (Ide et al., 2010)) requires substantial man- ual curation and analysis effort and does not scale up eas- ily; on the other hand, easy “opportunistic” data collection (Kennedy, 1998), where as much available material is col- lected from the Internet as possible, usually means little or no manual annotation or inspection of corpus contents for genre balance or other distributional properties, and at most the addition of relatively reliable automatic annota- tions such as part of speech tags and lemmatization (e.g. the WaCky corpora (Baroni et al., 2009)), or sometimes out- of-the-box, automatic dependency parsing (see the COW corpora, (Sch¨afer, 2015)). This divide leads to a number of problems: 1. The study of systematic genre variation is limited to surface textual properties and cannot use complex an- notations, unless it is confined to very limited data 2. For large resources, NLP quality is low since opportunistically-acquired content often deviates sub- stantially from the language found in training data used by out-of-the-box tools 3. Large open data sets for training NLP tools on com- plex phenomena (e.g. coreference resolution, discourse parsing) are unavailable1 In this paper, we attempt to walk a middle path, combining some of the best features of corpus data harvested from the web – size, open licenses, lexical diversity – and data col- lected using a carefully defined sampling frame (cf. Hundt (2008)), which allows for more interpretable inferences as well as the application of tools specially prepared for the 1The largest datasets for these tasks (e.g. RST Discourse Tree- bank (Carlson et al., 2003), OntoNotes (Weischedel et al., 2012)), contain between 200K and 1.6 million words and are only available for purchase from the LDC. target domain(s). In addition, we focus on making discourse- level annotations available, including complete nested entity recognition, coreference resolution, and discourse parsing, while striving for …

Content truncated to 500 words