Geographically-Balanced Gigaword Corpora for 50 Language Varieties

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2528–2536 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2528 Geographically-Balanced Gigaword Corpora for 50 Language Varieties Jonathan Dunn, Benjamin Adams University of Canterbury Christchurch, New Zealand {jonathan.dunn, benjamin.adams}@canterbury.ac.nz Abstract While text corpora have been steadily increasing in overall size, even very large corpora are not designed to represent global population demographics. For example, recent work has shown that existing English gigaword corpora over-represent inner-circle varieties from the US and the UK (Dunn, 2019b). To correct implicit geographic and demographic biases, this paper uses country-level population demographics to guide the construction of gigaword web corpora. The resulting corpora explicitly match the ground-truth geographic distribution of each language, thus equally representing language users from around the world. This is important because it ensures that speakers of under-resourced language varieties (i.e., Indian English or Algerian French) are represented, both in the corpora themselves but also in derivative resources like word embeddings. Keywords: corpus building, geo-referenced corpus, web as corpus, language mapping, under-resourced language varieties 1. Under-Represented Populations Where does digital language data come from and how well does it match up with the geographic distribution of hu- man populations? This is an important question because NLP now depends on large text corpora (Baroni et al., 2009; Goldhahn et al., 2012; Majliˇs and ˇZabokrtsk´y, 2012; Benko, 2014) that are derived from digital sources like web pages and social media. This paper raises two more cog- nate questions. First, are there populations that existing cor- pora fail to represent? Second, is there a significant differ- ence between regional varieties of languages, so that, e.g., a model trained on American English would work poorly on Nigerian English (Davies and Fuchs, 2015; Cook and Brinton, 2017)? If the answer to both of these questions is yes, it follows that NLP needs to switch to geographically- balanced datasets. This paper derives such geographically- balanced gigaword corpora from the 423 billion word Cor- pus of Global Language Use (Dunn, 2020). The result- ing family of corpora, GeoWAC, is evaluated against the CoNLL 2017 Shared Task data (Ginter et al., 2017) to de- termine the differences between otherwise comparable bal- anced and unbalanced corpora. In Section 2 we review recent work showing that there are strong geographic effects present in digital language sources. Then in Section 3 we describe the collection and cleaning methods used to produce the GeoWAC corpus family. In Section 4 we develop a population-based sam- pling method that is used to adjust the amount of data drawn from each country. Section 5 describes the geographic dis- tribution of the corpora that this sampling method produces. Section 6 evaluates the GeoWAC corpora against a baseline dataset that is not collected with geographic information, using both a corpus similarity evaluation and a word em- bedding evaluation. The primary contribution of this paper is to systematically remove the geographic biases that are present in existing gigaword corpora and to evaluate …

Content truncated to 500 words