Does Corpus Quality Really Matter for Low-Resource Languages?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7383 - 7390 December 7-11, 2022 ©2022 Association for Computational Linguistics Does Corpus Quality Really Matter for Low-Resource Languages? Mikel Artetxe1 Itziar Aldabe2 Rodrigo Agerri2 Olatz Perez-de-Viñaspre2 Aitor Soroa2 1Meta AI 2HiTZ Center, University of the Basque Country (UPV/EHU) artetxe@meta.com {itziar.aldabe,rodrigo.agerri,olatz.perezdevinaspre,a.soroa}@ehu.eus Abstract The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl. While prior work has identi- fied major issues on the quality of these datasets (Kreutzer et al., 2021), it is not clear how this impacts downstream performance. Taking rep- resentation learning in Basque as a case study, we explore tailored crawling—manually identi- fying and scraping websites with high-quality content—as an alternative to filtering Common- Crawl. Our new corpus, called EusCrawl, is similar in size to the Basque portion of popular multilingual corpora like CC100 and mC4, yet it has a much higher quality according to native annotators. For instance, 66% of documents are rated as high-quality for EusCrawl, in contrast with < 33% for both mC4 and CC100. Nev- ertheless, we obtain similar results on down- stream NLU tasks regardless of the corpus used for pre-training. Our work suggests that NLU performance in low-resource languages is not primarily constrained by the quality of the data, and other factors like corpus size and domain coverage can play a more important role. 1 Introduction Large-scale pre-training has resulted in a paradigm shift in NLP (Bommasani et al., 2021). While re- cent progress has been primarily driven by scaling up on model size and compute, both data quan- tity and quality have been shown to play a critical role (Kaplan et al., 2020; Rae et al., 2022). Nev- ertheless, existing efforts on data curation have primarily focused on English, and recent work on multilingual pre-training has relied on automati- cally filtered versions of CommonCrawl. For in- stance, XLM-R was trained on CC100 (Conneau et al., 2020), mT5 was trained on mC4 (Xue et al., 2021), and XGLM was trained on CC100-XL (Lin et al., 2021), which were all obtained by running language identification on several CommonCrawl snapshots and filtering through language-agnostic approaches. Unfortunately, Kreutzer et al. (2021) identified major issues on the quality of such multi- lingual datasets, ranging from language identifica- tion errors to boilerplate and non-linguistic content. However, the practical impact of these issues has not been studied, and it is unclear the extent to which higher-quality data could lead to better per- formance in low-resource languages. In this paper, we take representation learning in Basque as a case study, and explore tailored crawl- ing (i.e., manually identifying and scraping web- sites with high-quality content) as an alternative to filtering CommonCrawl. We introduce EusCrawl, a new corpus for Basque comprising 12.5M docu- ments from 33 websites with Creative Commons content. EusCrawl is similar in size to the Basque portion of CC100 and mC4, but it has substantially less issues and a higher perceived quality according to our blind audit with native annotators. …

Content truncated to 500 words