Building the Macedonian-Croatian Parallel Corpus

Building the Macedonian-Croatian Parallel Corpus Ines Cebović, Marko Tadić Faculty of Humanities and Social Sciences, University of Zagreb Ivana Lučića 3, 10000 Zagreb, Croatia ines.cebovic@gmail.com, marko.tadic@ffzg.hr Abstract In this paper we present the newly created parallel corpus of two under-resourced languages, namely, Macedonian-Croatian Parallel Corpus (mk-hr_pcorp) that has been collected during 2015 at the Faculty of Humanities and Social Sciences, University of Zagreb. The mk-hr_pcorp is a unidirectional (mk→hr) parallel corpus composed of synchronic fictional prose texts received already in digital form with over 500 Kw in each language. The corpus was sentence segmented and provides 39,735 aligned sentences. The alignment was done automatically and then post-corrected manually. The alignments order was shuffled and this enabled the corpus to be available under CC-BY license through META-SHARE. However, this prevents the research in language units over the sentence level. Keywords: written corpus, parallel corpus, Macedonian, Croatian 1. Introduction Parallel corpora are needed more and more as the source of bilingual and/or multilingual data for different research tasks in linguistics and Natural Language Processing. The research in contrastive linguistics, bilingual lexicography, translation studies, up to a traning of statistical machine translation models are just few of them. Today in the majority of parallel corpora it is usual that at least one of languages is a well-resourced language, while parallel corpora with two (or more) under-resourced languages are rather rare. In this paper we present the newly created parallel corpus of two under-resourced languages, namely, Macedonian-Croatian Parallel Corpus (mk-hr_pcorp) that has been collected during 2015 in collaboration between two departments of the Faculty of Humanities and Social Sciences, University of Zagreb: Department of South-Slavic Languages and Department of Linguistics. The paper is composed as follows: in Section 2. we mention the previous work on parallel corpora that include these two languages. In Section 3. we describe corpus parameters of mk-hr_pcorp and in Section 4 we provide information about the corpus composition. The Section 5 describes the processing and in Section 6 we give conclusion and mention possible future development. 2. Previous work To the best of our knowledge, there are no parallel texts collected for this language pair in particular, but there are larger multilingual text collections published earlier out of which a parallel Macedonian-Croatian bitexts could be extracted: • SETimes corpus, published in OPUS (Tiedemann, 2009) 1 and (Tyers & Alperen, 2010), which was later corrected and cleaned up 1 http://opus.lingfil.uu.se/ SETIMES.php (Ljubešić, 2013) 2 of residual HTML code, wrong language identification and missing diacritics3; • OpenSubs (Tiedemann, 2012), issues 20124 and 20135. These large multilingual text collections were created by automatical crawling or conversion from digital texts available in different input formats and then processed for noise removal. Although numerous automatic checking procedures were applied, still within these text collections a substantial amount of errors and noise can be found, starting with the wrong language ISO-codes attachment, up to the systemic errors in encoding, i.e. missing diacritics, or inaccurate alignments. The mk-hr_pcorp differs from other existing parallel text collections for this …

Content truncated to 500 words