A Geometric View on Bilingual Lexicon Extraction from Comparable Corpora

A Geometric View on Bilingual Lexicon Extraction from Comparable Corpora E. Gaussier†, J.-M. Renders†, I. Matveeva∗, C. Goutte†, H. D´ejean† †Xerox Research Centre Europe 6, Chemin de Maupertuis — 38320 Meylan, France Eric.Gaussier@xrce.xerox.com ∗Dept of Computer Science, University of Chicago 1100 E. 58th St. Chicago, IL 60637 USA matveeva@cs.uchicago.edu Abstract We present a geometric view on bilingual lexicon extraction from comparable corpora, which allows to re-interpret the methods proposed so far and iden- tify unresolved problems. This motivates three new methods that aim at solving these problems. Empir- ical evaluation shows the strengths and weaknesses of these methods, as well as a significant gain in the accuracy of extracted lexicons. 1 Introduction Comparable corpora contain texts written in differ- ent languages that, roughly speaking, ”talk about the same thing”. In comparison to parallel corpora, ie corpora which are mutual translations, compara- ble corpora have not received much attention from the research community, and very few methods have been proposed to extract bilingual lexicons from such corpora. However, except for those found in translation services or in a few international organ- isations, which, by essence, produce parallel docu- mentations, most existing multilingual corpora are not parallel, but comparable. This concern is re- flected in major evaluation conferences on cross- language information retrieval (CLIR), e.g. CLEF1, which only use comparable corpora for their multi- lingual tracks. We adopt here a geometric view on bilingual lex- icon extraction from comparable corpora which al- lows one to re-interpret the methods proposed thus far and formulate new ones inspired by latent se- mantic analysis (LSA), which was developed within the information retrieval (IR) community to treat synonymous and polysemous terms (Deerwester et al., 1990). We will explain in this paper the moti- vations behind the use of such methods for bilin- gual lexicon extraction from comparable corpora, and show how to apply them. Section 2 is devoted to the presentation of the standard approach, ie the ap- proach adopted by most researchers so far, its geo- metric interpretation, and the unresolved synonymy 1http://clef.iei.pi.cnr.it:2002/ and polysemy problems. Sections 3 to 4 then de- scribe three new methods aiming at addressing the issues raised by synonymy and polysemy: in sec- tion 3 we introduce an extension of the standard ap- proach, and show in appendix A how this approach relates to the probabilistic method proposed in (De- jean et al., 2002); in section 4, we present a bilin- gual extension to LSA, namely canonical correla- tion analysis and its kernel version; lastly, in sec- tion 5, we formulate the problem in terms of prob- abilistic LSA and review different associated simi- larities. Section 6 is then devoted to a large-scale evaluation of the different methods proposed. Open issues are then discussed in section 7. 2 Standard approach Bilingual lexicon extraction from comparable cor- pora has been studied by a number of researchers, (Rapp, 1995; Peters and Picchi, 1995; Tanaka and Iwasaki, 1996; Shahzad et al., 1999; Fung, 2000, among others). Their work relies on the assump- tion …

Content truncated to 500 words