A Cheap and Fast Way to Build Useful Translation Lexicons

A cheap and fast way to build useful translation lexicons Dan TUFIS Romanian Academy Centre for Artificial Intelligence 13, “13 Septembrie” Bucharest, ROMANIA, RO-74311 tufis@racai.ro Abstract The paper presents a statistical approach to automatic building of translation lexicons from parallel corpora. We briefly describe the pre-processing steps, a baseline iterative method, and the actual algorithm. The evaluation for the two algorithms is presented in some detail in terms of precision, recall and processing time. We conclude by briefly presenting some of our applications of the multilingual lexicons extracted by the method described herein. Introduction The scientific and technological advancement in many domains is a constant source of new term coinage and therefore keeping up with multilingual lexicography in such areas is very difficult unless computational means are used. Translation lexicons, based on translation equivalence relation are lexical knowledge sources, which can be extracted from parallel texts (even from comparable texts), with very limited human resources. The translation lexicons appear to be quite different from the corresponding printed lexicons, meant for the human users. There are well known reasons for these differences and we will not discuss the issue here, but exactly these differences make them very useful (in spite of inherent noise content) in many computer-based applications. We will discuss some of our experiments based on automatically extracted multilingual lexicons. Most modern approaches to automatic extraction of translation equivalents rely on statistical techniques and roughly fall into two categories. The hypotheses-testing methods such as Gale and Church (1991), Smadja et al. (1996), Tiedmann (1998), Ahrenberg (2000), Melamed (2001) etc. use a generative device that produces a list of translation equivalence candidates (TECs), extracted from corresponding segments of the parallel texts (translation units-TU), each of them being subject to an independence statistical test. The TECs that show an association measure higher than expected under the independence assumption are assumed to be translation-equivalence pairs (TEPs). The TEPs are extracted independently one of another and therefore the process might be characterized as a local maximization (greedy) one. The estimating approach such as Brown et al. (1993), Kay and Röscheisen (1993), Kupiec (1993), Hiemstra (1997) etc. is based on building from data a statistical bitext model, the parameters of which are to be estimated according to a given set of assumptions. The bitext model allows for global maximization of the translation equivalence relation, considering not individual translation equivalents but sets of translation equivalents (sometimes called assignments). There are pros and cons for each type of approach, some of them discussed in Hiemstra (1997). Our translation equivalents extraction process may be characterized as a “hypotheses testing” approach and does not need a pre-existing bilingual lexicon for the considered languages. If such a lexicon exists it can be used to eliminate spurious candidate translation equivalence pairs and thus to speed up the process and increase its accuracy. 1 Assumptions, preprocessing and a baseline There are several underlying assumptions one may consider in keeping the computational complexity of a translation lexicon extraction algorithm as low as possible. …

Content truncated to 500 words