Bilingual lexicon extraction from comparable corpora for closely related languages

Proceedings of Recent Advances in Natural Language Processing, pages 125–131, Hissar, Bulgaria, 12-14 September 2011. Bilingual Lexicon Extraction from Comparable Corpora for Closely Related Languages Darja Fiˇser Faculty of Arts, Univeristy of Ljubljana darja.fiser@ff.uni-lj.si Nikola Ljubeˇsi´c Faculty of Humanities and Social Sciences, University of Zagreb nikola.ljubesic@ffzg.hr Abstract In this paper we present a knowledge-light approach to extract a bilingual lexicon for closely related languages from compara- ble corpora. While in most related work an existing dictionary is used to trans- late context vectors, we take advantage of the similarities between languages in- stead and build a seed lexicon from words that are identical in both languages and then further extend it with context-based cognates and translations of the most fre- quent words. We also use cognates for reranking translation candidates obtained via context similarity and extract transla- tion equivalents for all content words, not just nouns as in most related work. The results are very encouraging, suggesting that other similar languages could bene- fit from the same approach. By enlarging the seed lexicon with cognates and trans- lations of the most frequent words and by cognate-based reranking of translation candidates we were able to improve the average baseline precision from 0.592 to 0.797 on the mean reciprocal rank for the ten top-ranking translation candidates for nouns, verbs and adjectives with a 46% re- call on the gold standard of 1000 random entries from a traditional dictionary. 1 Introduction Most cross-lingual NLP applications require bilin- gual lexicons but their compilation is still a ma- jor bottleneck in computational linguistics. Au- tomatic extraction of bilingual lexicons is typi- cally performed on parallel corpora (Och and Ney, 2000) but they exist only for a limited number of language pairs and domains and it is often imprac- tical or even impossible to build one from scratch. This is why an alternative approach has become popular in recent years. It relies on texts in two languages which are not parallel but comparable (Fung, 1998; Rapp, 1999) and therefore easier to compile, especially from the increasingly rich web data (Xiao and McEnery, 2006). The approach re- lies on the assumption that the term and its transla- tion appear in similar contexts (Fung, 1998; Rapp, 1999). This means that the translation of a source word can be found by identifying a target word which has the most similar context vector in a comparable corpus. However, a direct compari- son of vectors in two different languages is not possible, which is why a dictionary is needed to first translate the features of source context vec- tors into the target language and compute simi- larity on those. But this step seems paradoxical: the very reason why we are applying the com- plex comparable corpus technique for extracting translation equivalents is the fact that we do not have a bilingual dictionary at our disposal in the first place. This issue has largely remained un- addressed in previous research, which is why we propose a knowledge-light approach that does not require …

Content truncated to 500 words