European Union Language Resources in Sketch Engine

European Union Language Resources in Sketch Engine Vít Baisa, Jan Michelfeit, Marek Medved’, Miloš Jakubíˇcek Masaryk University, Czech Republic, Brno Lexical Computing Ltd, United Kingdom, Brighton {vit.baisa,jan.michelfeit,marek.medved,milos.jakubicek}@sketchengine.co.uk Abstract Several parallel corpora built from European Union language resources are presented here. They were processed by state-of-the-art tools and made available for researchers in the Sketch Engine corpus management system. A completely new resource is introduced: EUR-Lex corpus, being one of the largest parallel corpus available at the moment, containing 840 million tokens of English and having the largest language pair (English-French) with more than 25 million aligned segments (paragraphs). Keywords: JRC-Acquis, DCEP, DGT-TM, Europarl, EUR-Lex, Sketch Engine, parallel corpus, word sketch, parallel concordance 1. Introduction The European Union is producing a large amount of valu- able multilingual textual data every day. To be able to use it in applications, for text analysis, terminology extrac- tion, full text search etc., it must be downloaded, converted into plain text, processed with suitable tools, aligned on sentence level and finally made available to researchers in some standard format. In this paper we describe our ex- perience with using several resources built from European Union’s (EU) multilingual resources, namely DCEP (Ha- jlaoui et al., 2014), DGT-TM (Steinberger et al., 2013) and Europarl (Koehn, 2005). We also describe a new multilingual “EUR-Lex corpus” containing more than 840 million tokens of English. To our knowledge, it is currently the largest parallel corpus built from European language resources. The corpus was downloaded from the official website of EUR-Lex1 which provides an access to up-to-date legal documents published by European Commission, European Parliament, national courts, Council of the European Union and other European institutions. The majority of recently added documents is translated into all official languages of EU making it a huge multilingual language resource. Corpus Tokens Types L Format JRC-Acquis 55,537,910 N/A 22 XML DCEP 118,046,857 513,000 23 TXT DGT-TM 74,365,007 342,340 24 TMX Europarl 60,741,877 139,217 21 XML EUR-Lex 839,745,466 2,416,841 24 various Table 1: Comparison of various EU corpora. All mentioned corpora are available for language re- searchers through the Sketch Engine corpus management system (Kilgarriff et al., 2014). EUR-Lex corpus is released in the form of gzipped archives containing a) documents with meta information in a flat XML format and b) align- ment files for all language pairs. The whole gzipped dataset is over 40 GB.2 1http://eur-lex.europa.eu 2To obtain the data, contact us or follow the instructions at https://www.sketchengine.co.uk/eur-lex Table 1 compares the mentioned language resources. JRC- Acquis 3.0 figures3 are there for comparison. “Tokens” is the number of tokens (words, numbers and punctuation) in the English parts of the corpora. “Types” is the num- ber of unique English word forms, i.e. the size of English lexicons, “L” column contains the number of languages in- cluded and “Format” states in which form the source data is available. Language Since ACQ CEP DGT EUR LEX Dutch 1958 35 96 63 60 777 French 1958 39 116 47 67 878 German 1958 32 98 58 55 732 …

Content truncated to 500 words