MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages

MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages Marta Ba˜n´on†, Miquel Espl`a-Gomis⋆, Mikel L. Forcada⋆, Cristian Garc´ıa-Romero⋆, Taja Kuzman‡, Nikola Ljubeˇsi´c‡, Rik van Noord♦, Leopoldo Pla Sempere⋆, Gema Ram´ırez-S´anchez†, Peter Rupnik‡, V´ıt Suchomel‡, Antonio Toral♦, Tobias van der Werff♦, Jaume Zaragoza† ‡Joˇzef Stefan Institute, †Prompsit, ♦Rijksuniversiteit Groningen, ⋆Universitat d’Alacant ‡{taja.kuzman,nikola.ljubesic,peter.rupnik}@ijs.si, vit.suchomel@sketchengine.eu †{mbanon,gramirez,jzaragoza}@prompsit.com ♦{r.i.k.van.noord,a.toral.ruiz,t.n.van.der.werff}@rug.nl ⋆{mespla,mlf,cgarcia,lpla}@dlsi.ua.es Abstract We introduce the project MaCoCu: Mas- sive collection and curation of monolin- gual and bilingual data: focus on under- resourced languages, funded by the Con- necting Europe Facility, which is aimed at building monolingual and parallel corpora for under-resourced European languages. The approach followed consists of crawl- ing large amounts of textual data from se- lected top-level domains of the Internet, and then applying a curation and enrichment pipeline. In addition to corpora, the project will release the free/open-source web crawl- ing and curation software used. 1 Introduction This paper describes the project MaCoCu: Massive collection and curation of monolingual and bilin- gual data: focus on under-resourced languages, funded by the Connecting Europe Facility in the 2020 CEF Telecom Call - Automated Translation (2020-EU-IA-0078).1 This project started on June 1, 2021, and will last for two years. It is aimed at building large and high-quality monolingual and parallel (with English) corpora for five under- resourced official EU languages: Maltese, Bulgar- ian, Slovenian, Croatian, and Icelandic;2 and for the languages of the five candidate states to become EU members: Turkish, Albanian, Macedonian, Mon- © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1https://ec.europa.eu/inea/connecting-eur ope-facility/cef-telecom/2020-eu-ia-0078 2Maltese and Icelandic were chosen since they are especially under-resourced official EU languages; Bulgarian, Slovenian and Croatian were chosen due to the interest of the consortium on South-Slavic languages, a decision that extends previous efforts in the Abu-MaTran project (Toral et al., 2015). tenegrin, and Serbian. Existing initiatives produc- ing similar corpora, such as Paracrawl (Ba˜n´on et al., 2020) or Oscar (Abadji et al., 2022) exploit existing resources such as Common Crawl3 or the Internet Archive.4 In contrast, our strategy consists in auto- matically crawling top-level domains (TLD) with the potential to contain substantial amounts of tex- tual data in the targeted languages,5 and then apply- ing a monolingual and a parallel curation pipelines on the downloaded data. This approach aims at obtaining more and higher-quality data than that available in existing compilations.6 One of the objectives of the project is to iden- tify data relevant for Digital Service Infrastructures (DSIs). Our corpora will be enriched with infor- mation about the relevance of the data collected for ten DISs: e-Health, e-Justice, Online Dispute Resolution, Europeana, Open Data Portal, Business Registers Interconnection System, e-Procurement, Safer Internet, Cybersecurity, and Electronic Ex- change of Social Security Information. 1.1 International consortium Four partners are involved in this project: Institut Joˇzef Stefan (Slovenia), Rijksuniversiteit Gronin- gen (Netherlands), Prompsit Language Engineer- ing S.L. (Spain), and Universitat d’Alacant (Spain; coordinator). The consortium has a strong back- ground in the task of building …

Content truncated to 500 words