Parallel Global Voices: a Collection of Multilingual Corpora with Citizen Media Stories

Parallel Global Voices: a Collection of Multilingual Corpora with Citizen Media Stories Prokopis Prokopidis, Vassilis Papavassiliou and Stelios Piperidis Institute for Language and Speech Processing Athena Research and Innovation Center, Athens, Greece {prokopis, vpapa, spip}@ilsp.gr Abstract We present a new collection of multilingual corpora automatically created from the content available in the Global Voices websites, where volunteers have been posting and translating citizen media stories since 2004. We describe how we crawled and processed this content to generate parallel resources comprising 302.6K document pairs and 8.36M segment alignments in 756 language pairs. For some language pairs, the segment alignments in this resource are the first open examples of their kind. In an initial use of this resource, we discuss how a set of document pair detection algorithms performs on the Greek-English corpus. Keywords: parallel corpus, independent news stories, document pair detection 1. Introduction This work describes Parallel Global Voices (PGV), a col- lection of parallel corpora created from the Global Voices1 group of websites. According to the Global Voices (GV) project description, GV “is a border-less, largely volunteer community of more than 1400 writers, analysts, online me- dia experts and translators”. The GV community has been reporting since 2004 on trending issues and stories pub- lished on social media and independent blogs in 167 coun- tries. In this paper we provide an overview of how we used this content to automatically generate an open set of parallel cor- pora that exhibit some interesting features as far as topics and language pairs are concerned. The rest of the paper is organized as follows. After discussing related work in Sec- tion 2, we report in Section 3 on how the Global Voices content was crawled, processed and aligned at document and sentence level. Section 4 provides details on the size and characteristics of monolingual and parallel sub-corpora in PGV and on format and availability. As an example of the potential use of the parallel corpora presented here, we use them to examine methods for the detection of parallel web pages and discuss results in Section 5. 2. Related Work Parallel corpora like PGV are important for a number of applications (Tiedemann, 2011) including SMT, induction of bilingual lexica and contrastive studies of language use. Tlaxcala (Toral, 2014) was the first publicly available col- lection of parallel and monolingual corpora acquired from independent news sources. The largest parallel corpus in this 15-language resource is English-Spanish with 66.8K sentence pairs. Rettinger et al. (2014) compiled a par- allel corpus of 300 English/Spanish/German GV articles, which they hand-annotated with semantic groundings of named entities and concepts to cross-lingual linked data ex- tracted from Wikipedia. Chahuneau et al. (2013) used an English-Swahili parallel corpus obtained by crawling GV and reported significant improvements in translation qual- 1https://globalvoices.org/ ity when translating to Swahili. Finally, a GV Malagasy- English parallel corpus, collected and aligned at sentence level by V. Chahuneau, is available from http://www. ark.cs.cmu.edu/global-voices/. This work is to the best of our knowledge the first that extracts parallel …

Content truncated to 500 words