Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4344–4355 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4344 Towards a Cleaner Document-Oriented Multilingual Crawled Corpus Julien Abadji1, Pedro Ortiz Suarez1,2, Laurent Romary1, Benoˆıt Sagot1 Inria1, Sorbonne Universit´e2. 2 rue Simone Iff, 75012 Paris1, 21 rue de l’´Ecole de m´edecine, 75006 Paris2. {julien.abadji, pedro.ortiz, laurent.romary, benoit.sagot}@inria.fr Abstract The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities. Keywords: Web corpus, Language Modeling, Common Crawl 1. Introduction The demand for large corpora has considerably in- creased in recent years with the advent of semi- supervised learning methods in Natural Language Pro- cessing (NLP), such as word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Mikolov et al., 2018), contextualized word representations (Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019) and more recently very large generative lan- guage models like GPT-3, T5, GPT-Neo (Raffel et al., 2020; Brown et al., 2020; Black et al., 2021). While there have been some recent efforts to manually cu- rate such corpora1 (Gao et al., 2020), the common ap- proach to collect large amounts of raw textual data still relies primarily on crawled web text (Ortiz Su´arez et al., 2019; Ortiz Su´arez et al., 2020; Xue et al., 2021; El-Kishky et al., 2020; Espl`a et al., 2019; Ba˜n´on et al., 2020; Gao et al., 2020), and although some of the initial concerns of using crawled data (Trinh and Le, 2018; Radford et al., 2019) have been addressed in recent years (Ortiz Su´arez et al., 2020; Martin et al., 2020) there a many concerns that still need to be tack- led (Caswell et al., 2020) specially for multilingual data (Caswell et al., 2021). In this demand for large raw textual corpora we can ob- serve a clear back and forth in the type of data used to pre-train these models. On one hand some authors have opted for highly curated or edited data like Wikipedia such as Al-Rfou’ et al. (2013) and Bojanowski et al. (2017) for static word embeddings, the 1B Word Benchmark (Chelba et al., 2014) for ELMo (Peters et al., 2018), and the BookCorpus (Zhu et al., 2015) and Wikipedia for BERT (Devlin et al., 2019). On the other …

Content truncated to 500 words