Automatic Bilingual Lexicon Extraction for a Minority Target Language

Automatic Bilingual Lexicon Extraction for a Minority Target Language* Eileen Pamela Tiu a, and Rachel Edita O. Roxas a aCollege of Computer Studies, De La Salle University-Manila 2401 Taft Avenue, Malate, Manila, Philippines roxasr@dlsu.edu.ph Abstract. An automated approach of extracting bilingual lexicon from comparable, non- parallel corpora was developed for a target language with limited linguistic resources. We combined approaches from previous researches which only concentrated on context extraction, clustering techniques, or usage of part of speech tags for defining the different senses of a word. The domain-specific corpora for the source language contain 381,553 English words, while the target language with minimal language resources contain 92,610 Tagalog word, with 4,817 and 3,421 distinct root words, respectively. Despite the use of limited amount of corpora (400k vs Sadat’s (2003) 39M word corpora) and seed lexicon (9,026 entries vs Rapp’s (1999) 16,380 entries), the evaluation yielded promising results. The 50 high and 50 low frequency words yielded 50.29% and 31.37% recall values, and 56.12% and 21.98% precision values, respectively, which are within the range of values from previous studies, 39 - 84.45% (Koehn et al., 2002 and Zhou et al., 2001). Ranking showed an improvement to overall F-measure from 7.32% to 10.65%. Keywords: Automatic lexicon extraction. 1. Introduction Automatic bilingual lexicon extraction is the automated process of acquiring bilingual word pairs from corpora in order to construct a lexicon. The product of this process, a bilingual lexicon (or a dictionary), is commonly used for Machine Translation, Natural Language Processing, and other linguistic usages. Since acquiring lexicon from parallel corpora already yields 99% accuracy (Rapp, 1999), this study focused into processing non-parallel corpora, specifically on comparable corpora. In addition, a lexicon is more helpful if it has the feature of grouping similar senses together. This feature is called word sense discrimination. In this study, the syntax and senses of the word are the contributing factors to discriminate words that have several meanings. Throughout the years, linguists and translators compile different types of lexicons manually from experts and automatically from corpora, which are collections of texts and other forms of writings. Parallel corpora refer to a source text and its translation into one or more target languages (Ahrenberg, et al., 1999). Parallel corpora are used for lexicon extraction due to the following characteristics (Fung, 1998): (1) Words have one sense per corpus; (2) Words have single translation per corpus; (3) No missing translations in the target document; (4) Frequencies of bilingual word occurrences are comparable; and (5) Positions of bilingual word occurrences are comparable. However, acquiring parallel corpora is labor intensive and time consuming. It is also unlikely that one can find parallel corpora in any given domain in electronic form especially for minority languages. This problem can be solved by using non-parallel corpora. Non-parallel corpora * Copyright 2007 by Eileen Pamela Tiu, and Rachel Edita O. Roxas 368 22nd Pacific Asia Conference on Language, Information and Computation, pages 368–376 come in two forms: comparable and non-comparable corpora. Comparable corpora are collections of …

Content truncated to 500 words