Two Years of Aranea: Increasing Counts and Tuning the Pipeline

Two Years of Aranea: Increasing Counts and Tuning the Pipeline Vladimír Benko Slovak Academy of Sciences, Ľ. Štúr Institute of Linguistics Panská 26, SK-81101 Bratislava, Slovakia E-mail: vladob@juls.savba.sk Abstract The Aranea Project is targeted at creation of a family of Gigaword web-corpora for a dozen of languages that could be used for teaching language- and linguistics-related subjects at Slovak universities, as well as for research purposes in various areas of linguistics. All corpora are being built according to a standard methodology and using the same set of tools for processing and annotation, which – together with their standard size and– makes them also a valuable resource for translators and contrastive studies. All our corpora are freely available either via a web interface or in a source form in an annotated vertical format. Keywords: web corpora, open-source and free tools 1. Introduction During the last decade, creation of web-derived corpora has been recognized as an effective way of obtaining language data in situations where building traditional corpora would be too costly or too slow (Baroni et al., 2004, 2009; Saharoff, 2006; Jakubíček et al., 2013; Schäfer & Bildhauer, 2013). The recently released open-source tools for this purpose made it possible (even for low-funded educational and research institutions in Central Europe) to undertake projects of creation large-scale web corpora. Within the framework of our Aranea project, a family of Gigaword web corpora for some two dozens of languages is being created. In our previous paper (Benko, 2014a), we presented the state of the Project after its first year, introduced the methodology and tools used, and demonstrated some results for the first eight languages processed (i.e., Dutch, English, French, German, Polish, Russian, Slovak and Spanish). In this paper, we would like to show the new developments within the Aranea family of corpora during the past year; summarize the results and discuss the possible future of the Project. 2. The Aranea Project The main reason why we decided to start our Project was the lack of suitable corpora that could be used for teaching purposes at our Universities. When starting our Project in Spring 2013, the main design decisions of were set as follows: • The corpora family will be ―Slovak-centric‖ (i.e., it will consist of languages used and/or taught in Slovakia and its neighbouring countries; • All corpora will bear ―language-neutral‖ (Latin) names ; • They will be designed as ―comparable‖ (i.e., the same size(s), methodology of processing and annotation); • Will be (freely) available via a web interface; • A compatible sketch grammar for the Sketch Engine will be written for each language. The second year of the Aranea Project was dedicated to completing the language list of the ―inner circle‖ (i.e., Czech, Hungarian and Ukrainian – the latter, however, without any annotation yet), and adding four foreign languages taught at our University (Chinese, Finnish, Italian, and Portuguese). For English and Russian, territory-specific corpora have also been created by restricting web crawling to national top-level domains of countries where English/Russian have an …

Content truncated to 500 words