Multilingual Open Text Release 1: Public Domain News in 44 Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2080–2089 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2080 Multilingual Open Text Release 1: Public Domain News in 44 Languages Chester Palen-Michel∗, June Kim∗, Constantine Lignos Michtom School of Computer Science Brandeis University {cpalenmichel, junekim, lignos}@brandeis.edu Abstract We present a Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001–2022 and collected from Voice of America’s news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published. Keywords: multilingual corpora, text data, low resource NLP, open access text 1. Introduction This work describes the first release of Multilin- gual Open Text (MOT), a collection of permis- sively licensed texts created with a goal of improv- ing the amount of high-quality text available for lower- resourced languages. MOT Release 1 consists of data collected from Voice of America (VOA) news websites. Our broader goal is a corpus of open access multilingual text, and we plan to include data from other sources in future re- leases. As part of the development of this corpus, we created infrastructure to continue to scrape new docu- ments as they are published in order to provide subse- quent releases with newly published and updated doc- uments. We have been using this infrastructure for sev- eral months to expand the corpus. The corpus contains documents in many different languages, many of which are lower-resourced. In this paper, we explain our process for collecting, fil- tering, and processing the data from VOA news web- sites in multiple languages and describe the resulting corpus. In Section 2, we motivate the need for this corpus and compare with similar lower-resourced lan- guage dataset creation efforts. In Section 3, we de- scribe the content of MOT. In Section 4, we detail our process for creating the corpus. Finally, in Section 5, we discuss limitations and future directions. The cor- pus is available via GitHub.1 2. Related Work A multilingual collection of unlabeled text can be use- ful for many tasks, especially for lower-resourced lan- guages with limited freely-available text. An unlabeled non-parallel corpus is typically the starting point for further annotation and dataset creation work. Much of *Equal contribution. 1https://github.com/bltlab/mot/ modern NLP relies on either pre-trained static or con- textual word embeddings; in either case, these meth- ods rely on large quantities of text data, which lower- resourced languages lack. Even with the existence of multilingual Transformer models, like multilingual BERT (Devlin et …

Content truncated to 500 words