Multext-East: Parallel and Comparable Corpora and Lexicons for Six Central and Eastern European Languages

Multext-East: Parallel and Comparable Corpora and Lexicons for Six Central and Eastern European Languages Ludmila DIMITROVA Institute of Mathematics and Informatics Sofia, Bulgaria ludmila@ling.math.acad.bg Nancy IDE Dept, of Computer Science, Vassar College Poughkeepsie, New York, USA ide@cs.vassar.edu Vladimir PETKEVIC Inst. of Theoretical and Computational Linguistics, Charles University Prague, Czech Republic Vladimir.Petkevic@ff.cuni.cz Tomaz ERJAVEC Institute Jozef Stefan Ljubljana, Slovenia tomaz.erjavec@ijs.si Heiki Jaan KAALEP Dept. of General Linguistics, University of Tartu Tartu, Estonia hkaalep@psych.ut.ee Dan TUFIS Romanian Academy Center for Artificial Intelligence Bucharest, Romania tuff s @racai.ro Abstract The EU Copernicus project Multext-East has created a multi-lingual corpus of text and speech data, covering the six languages of the project: Bulgarian, Czech, Estonian, Hungarian, Romanian, and Slovene. In addition, wordform lexicons for each of the languages were developed. The corpus includes a parallel component consisting of Orweli's Nineteen Eighty-Four, with versions in all six languages tagged for part-of-speech and aligned to English (also tagged for POS). We describe the encoding format and data architecture designed especially for this corpus, which is generally usable for encoding linguistic corpora. We also describe the methodology for the development of a harmonized set of morphosyntactic descriptions (MSDs), which builds upon the scheme for western European languages developed within the EAGLES project. We discuss the special concerns for handling the six project languages, which cover three distinct language families. Introduction In order to provide resources to enable the efficient extraction of quantitative and qualitative information from corpora, several corpus development and distribution efforts have been recently established. However, few corpora exist for Central and Eastern European (CEE) languages, and corpus-processing tools that take into account the specific characteristics of these languages are virtually non-existent. The Multext-East Copernicus project ~ (Erjavec, et al., 1997) was a spin-off of the LRE project Multext 2 (Ide and V4ronis, 1994) intended to fill these gaps by developing significant resources for six CEE languages (Bulgarian, Czech, Estonian, Hungarian, Romanian, Slovene) that follow a consistent and principled encoding format and are maximally suited to easy processing by corpus-handling tools. To this end, Multext-East developed a corpus of parallel and comparable texts for the six CEE project languages, together with wordform lexicons and other language-specific resources. In the following sections we briefly describe the Multext-East corpora (text, speech) and the Multext-East lexicons and language-specific resources, 1 The Multext-East corpora 1.1 Encoding format Based on the principle that its corpus encoding format should be standardized and homogeneous both for interchange and for facilitating open- ended retrieval tasks, Multext-East adopted the l http://nl.ijs.si/ME 2http://www.lpl.univ-aix.fr/ projects/Multext/ 315 Corpus Encoding Standard (CES) 3 (Ide, 1998), which has been developed to be optimally suited for use in language engineering and corpus- based work. The CES is an application of SGML (ISO-8879, Standard Generalized Markup Language) and is based on the TEl Guidelines for Electronic Text Encoding and Interchange. In addition to providing encoding conventions for elements relevant to corpus-based work, the CES provides a data architecture for linguistic corpora and their annotations. Each corpus component, comprising a single text …

Content truncated to 500 words