Improving Corpus Comparability for Bilingual Lexicon Extraction from Comparable Corpora

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 644–652, Beijing, August 2010 Improving Corpus Comparability for Bilingual Lexicon Extraction from Comparable Corpora Bo Li, Eric Gaussier Laboratoire d’Informatique de Grenoble (LIG) Universit´e de Grenoble firstname.lastname@imag.fr Abstract Previous work on bilingual lexicon extrac- tion from comparable corpora aimed at finding a good representation for the usage patterns of source and target words and at comparing these patterns efficiently. In this paper, we try to work it out in an- other way: improving the quality of the comparable corpus from which the bilin- gual lexicon has to be extracted. To do so, we propose a measure of comparabil- ity and a strategy to improve the qual- ity of a given corpus through an iterative construction process. Our approach, be- ing general, can be used with any existing bilingual lexicon extraction method. We show here that it leads to a significant im- provement over standard bilingual lexicon extraction methods. 1 Introduction Bilingual dictionaries are an essential resource in many multilingual natural language process- ing (NLP) tasks such as machine translation (Och and Ney, 2003) and cross-language information retrieval (CLIR) (Ballesteros and Croft, 1997). Hand-coded dictionaries are of high quality, but expensive to build and researchers have tried, since the end of the 1980s, to automatically extract bilingual lexicons from parallel corpora (see (Chen, 1993; Kay and R¨oscheisen, 1993; Melamed, 1997a; Melamed, 1997b) for early work). Parallel corpora are however difficult to get at in several domains, and the majority of bilingual collections are comparable and not par- allel. Due to their low cost of acquisition, sev- eral researchers have tried to exploit such com- parable corpora for bilingual lexicon extraction (Fung and McKeown, 1997; Fung and Yee, 1998; Rapp, 1999; D´ejean et al., 2002; Gaussier et al., 2004; Robitaille et al., 2006; Morin et al., 2007; Yu and Tsujii, 2009). The notion of comparability is however a loose one, and comparable corpora range from lowly comparable ones to highly com- parable ones and parallel ones. For data-driven NLP techniques, using better corpora often leads to better results, a fact which should be true for the task of bilingual lexicon extraction. This point has largely been ignored in previous work on the subject. In this paper, we develop a well-founded strategy to improve the quality of a comparable corpus, so as to improve in turn the quality of the bilingual lexicon extracted. To do so, we first pro- pose a measure of comparability which we then use in a method to improve the quality of the ex- isting corpus. The remainder of the paper is organized as fol- lows: Section 2 introduces the experimental mate- rials used for the different evaluations; compara- bility measures are then presented and evaluated in Section 3; in Section 4, we detail and evaluate a strategy to improve the quality of a given corpus while preserving its vocabulary; the method used for bilingual lexicon extraction is then described and evaluated in Section 5. Section 6 …

Content truncated to 500 words