Know thy Corpus! Robust Methods for Digital Curation of Web corpora

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2453–2460 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2453 Know thy corpus! Robust methods for digital curation of Web corpora Serge Sharoff Centre for Translation Studies, University of Leeds, LS2 9JT, UK s.sharoff@leeds.ac.uk Abstract This paper proposes a novel framework for digital curation of Web corpora in order to provide robust estimation of their parameters, such as their composition and the lexicon. In recent years language models pre-trained on large corpora emerged as clear winners in numerous NLP tasks, but no proper analysis of the corpora which led to their success has been conducted. The paper presents a procedure for robust frequency estimation, which helps in establishing the core lexicon for a given corpus, as well as a procedure for estimating the corpus composition via unsupervised topic models and via supervised genre classification of Web pages. The results of the digital curation study applied to several Web-derived corpora demonstrate their considerable differences. First, this concerns different frequency bursts which impact the core lexicon obtained from each corpus. Second, this concerns the kinds of texts they contain. For example, OpenWebText contains considerably more topical news and political argumentation in comparison to ukWac or Wikipedia. The tools and the results of analysis have been released. Keywords: Validation of language resources, Text analytics, Language Modelling, Digital curation 1. Introduction The vast amount of text data available from the Web pro- vides a very good window for looking into a lot of language at once (Sinclair, 1991), which is extremely useful for de- veloping language resources. For example, unlabelled data from the Web are often used to pre-train embeddings for fur- ther downstream applications. Many pre-trained language models, such as ELMO, BERT or GPT, have been produced by taking Web resources, such as the 1B Word Benchmark for ELMO, Wikipedia with the Toronto Book Corpus for BERT or WebText for GPT. However, the impact of using a particular Web-derived corpus on the outcomes of pre- training is not clear, since corpora obtained from the Web lack curated categories, while it is known that such factors as the domain and quality of the training set are crucial to the performance of the model. Digital curation of Web re- sources makes them similar in spirit to what has been used in traditional corpora, such as the BNC, as this allows esti- mation of its implications as well as drawing comparison to other Web corpora. One such issue is obtaining reliable estimates of the proba- bility of occurrences for words, n-grams and linguistic con- structions, since raw frequency counts are prone to bursts. Adam Kilgarriff referred to this as a "whelk" problem (Kil- garriff, 1997): if a text is about whelks, no matter how infrequent this word is in the rest of the corpus, it’s likely to be in nearly every sentence in this text. If a corpus has a small bias towards texts of a particular kind, …

Content truncated to 500 words