Producing Monolingual and Parallel Web Corpora at the Same Time - SpiderLing and Bitextor’s Love Affair

Producing Monolingual and Parallel Web Corpora at the Same Time – SpiderLing and Bitextor’s Love Affair Nikola Ljubeˇsi´c,∗ Miquel Espl`a-Gomis,† Antonio Toral,§ Sergio Ortiz-Rojas,‡ Filip Klubiˇcka∗ ∗Dept. of Information and Communication Sciences, †Dept. de Llenguatges i Sistemes Inform`atics, University of Zagreb, Zagreb (Croatia) Universitat d’Alacant, Alacant (Spain) nljubesi@ffzg.hr mespla@dlsi.ua.es fklubicka@ffzg.hr §ADAPT Centre, School of Computing, ‡Prompsit Language Engenering, Dublin City University, Dublin (Ireland) Elx (Spain) atoral@computing.dcu.ie sergio@prompsit.com Abstract This paper presents an approach for building large monolingual corpora and, at the same time, extracting parallel data by crawling the top-level domain of a given language of interest. For gathering linguistically relevant data from top-level domains we use the SpiderLing crawler, modified to crawl data written in multiple languages. The output of this process is then fed to Bitextor, a tool for harvesting parallel data from a collection of documents. We call the system combining these two tools Spidextor, a blend of the names of its two crucial parts. We evaluate the described approach intrinsically by measuring the accuracy of the extracted bitexts from the Croatian top-level domain .hr and the Slovene top-level domain .si, and extrinsically on the English–Croatian language pair by comparing an SMT system built from the crawled data with third-party systems. We finally present parallel datasets collected with our approach for the English–Croatian, English–Finnish, English–Serbian and English–Slovene language pairs. Keywords: crawling, top-level domain, monolingual corpus, parallel corpus 1 Introduction Parallel data are one of the most important linguis- tic resources for cross-lingual natural language process- ing (Melamed, 2001). Parallel corpora consist of collec- tions of texts in different languages which are mutual trans- lations. This resource is specially relevant in the field of statistical machine translation (SMT), where parallel cor- pora are used to learn translation models automatically. The growing interest in SMT in the last decades has increased the demand of parallel corpora and, as a consequence, new strategies have been proposed to collect such data. Many sources of bitexts have been identified; some examples are: • texts from multilingual institutions, such as the Hansards corpus (Roukos et al., 1995) or the Europarl corpus (Koehn, 2005); • translations of software interfaces and documentation, such as KDE4 and OpenOffice (Tiedemann, 2009); or • news translated into different languages, such as the SETimes corpus (Ljubeˇsi´c, 2009), or the News Com- mentaries corpus (Bojar et al., 2013). However, one of the most obvious sources for collecting parallel data is the Internet. On the one hand, most of the sources already mentioned are currently available on the Web. In addition to this, it is worth noting that many web- sites are available in several languages and this translated content is another useful source of parallel data. There- fore, a considerable scientific effort has been put during the last years in order to exploit the web as a source to auto- matically acquire new parallel data (see Section 2). Some examples of corpora built from multilingual web pages are the Tourism English–Croatian Parallel Corpus 2.01 (Toral et al., 2014) or the …

Content truncated to 500 words