MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages

MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages Marta Ba˜n´on†, M˘alina Chichir˘au♦, Miquel Espl`a-Gomis⋆, Mikel L. Forcada⋆, Aar´on Galiano-Jim´enez⋆, Taja Kuzman‡, Nikola Ljubeˇsi´c‡, Rik van Noord♦, Leopoldo Pla Sempere⋆, Gema Ram´ırez-S´anchez†, Peter Rupnik‡, V´ıt Suchomel‡, Antonio Toral♦, Jaume Zaragoza† ‡Joˇzef Stefan Institute, †Prompsit, ♦Rijksuniversiteit Groningen, ⋆Universitat d’Alacant ‡{taja.kuzman,nikola.ljubesic,peter.rupnik}@ijs.si, vit.suchomel@sketchengine.eu †{mbanon,gramirez,jzaragoza}@prompsit.com ♦{r.i.k.van.noord,a.toral.ruiz,m.chichirau}@rug.nl ⋆{mespla,mlf,cgarcia,lpla}@dlsi.ua.es Abstract We present the most relevant results of the project MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages in its second year. Parallel and monolingual cor- pora have been produced for eleven low- resourced European languages by crawling large amounts of textual data from selected top-level domains of the Internet; both hu- man and automatic evaluation show its use- fulness. In addition, several large language models pretrained on MaCoCu data have been published, as well as the code used to collect and curate the data. 1 Introduction This paper describes the main outcomes of the project MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under- resourced languages (Ba˜n´on et al., 2022), span- ning from June 2021 to July 2023. MaCoCu is aimed at building large and high-quality monolin- gual and parallel (with English) corpora for ten low- resourced European languages (see Table 1). The international consortium behind this project con- sists of four partners: Joˇzef Stefan Institute (Slove- nia), Rijksuniversiteit Groningen (Netherlands), Prompsit Language Engineering S.L. (Spain), and Universitat d’Alacant (Spain; coordinator). Other existing initiatives, such as Paracrawl1 or Oscar2 exploit existing resources such as Common Crawl3 or the Internet Archive.4 Our strategy con- © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1https://paracrawl.eu/ 2https://oscar-project.org/ 3https://commoncrawl.org/ 4https://archive.org/ sists in automatically crawling top-level domains (TLD), potentially containing substantial amounts of text in the targeted languages,5 and then apply- ing a monolingual and a parallel curation pipelines. The evaluation of the first data release (van Noord et al., 2022a) confirms the usefulness of these data for different natural-language processing tasks. 2 Collected corpora Monolingual and parallel corpora are built from crawled data by applying a thorough cleaning process, including noise fixing/filtering and re- moval of near-duplicate/boilerplate text. Corpora are then automatically annotated with: (a) doc- ument and paragraph IDs; (b) language variety (e.g. British/American English); (c) document- level affinity to DSIs identified through domain modelling (van Noord et al., 2022b); (d) personal information; and (e) identification of translated text: either human or machine translations (only for par- allel corpora). Table 1 shows the size of the corpora for the second data release, published in April 2023. 2.1 Data evaluation To the date, evaluation only covers the seven lan- guages included in the first data release of the ac- tion, made public in April of 2022. Mono-lingual A set of pre-trained language mod- els (LMs)6 has been built and released for Icelandic, Maltese and Bulgarian/Macedonian by continuing the training of multilingual XLM-RoBERTa-large (Conneau et al., 2020) using only MaCoCu data for all languages. These …

Content truncated to 500 words