Slovene–English Datasets for MT

Slovene-English Datasets for MT ����� �  � Dept. of Intelligent Systems    �!#"%$'&�()!�*�+,!�*.-/" Ljubljana, Slovenia Abstract Advances in machine translation are becoming increasingly dependent on the availability of large scale language resources, in particular parallel corpora. The talk presents Slovene-English language resources that were developed as datasets for translation studies and machine learning programs. Three parallel datasets are introduced: the MULTEXT-East multilingual word-annotated corpus, the IJS-ELAN Slovene-English parallel corpus, and the Concede English-Slovene dictionary fragment and lexical database. 1. Introduction While machine translation systems will, for a while to come, utilise hand-crafted lexica and rules, there is an increasingly important strand of research which attempts to develop methods that can automatically induce lexica and rules from language data, in particular from bilingual parallel language corpora. Such corpora must be developed separately for each language pair; it adds to their usability if they are additionally marked up for linguistic information, encoded in a standard manner and made widely available. The value of aligned corpora is well attested in practice, with several recent European projects devoted to producing them, e.g. MLCC [2] (nine EU languages), Crater [3] (Spanish, French and English), or ENPC [4] (English-Norwegian). Slovene is a South-Slavic language with approximately 2 million speakers. The language is characterised, as other Slavic languages, with a complex inflectional system and free word order. So far, no serious attempts have been made to develop MT systems or dedicated MT resources for Slovene. In the last few years, however, various EU projects in language technologies have enabled the development of parallel Slovene language resources, which could prove useful for experiments in MT resource acquisition. The parallel resources developed at our institute share a number of common characteristics. So far, we have concentrated on the Slovene-English language pair, as this combination can prove the most immediately useful. All the resources have been encoded in SGML, in particular in various parameterisation of the Text Encoding Initiative Guidelines, TEI P3 [5]. Such a standardised encoding makes the resources more portable between platforms and applications and ensures their longevity. Finally, we have made every effort to ensure easy access to the resources; any restrictions on their availability stem only from copyright on the original data. All this makes the datasets suitable for reuse for a wide variety of target applications. In the paper we concentrate on three datasets. Section 2. introduces the multilingual MULTEXT-East 100.000 word morphosyntactically annotated corpus and associated resources. We discuss the new release of the corpus, which has been significantly corrected and re-encoded from the original publication in 1998. Section 3. presents the Slovene- English IJS-ELAN 1 million word corpus, specifically targeted as a dataset for experiments in automatic bi-lingual terminology extraction. We discuss the current state of the corpus, and mention the forthcoming new version, which will be word-annotated for morphosyntax. Section 4. mentions the English-Slovene dictionary fragment, which we have converted into a lexical database. Finally, section 5. gives some conclusions and directions for further research. 2. The MULTEXT-East dataset …

Content truncated to 500 words