Compiling a Massive, Multilingual Dictionary via Probabilistic Inference

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 262–270, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Compiling a Massive, Multilingual Dictionary via Probabilistic Inference Mausam Stephen Soderland Oren Etzioni Daniel S. Weld Michael Skinner* Jeff Bilmes University of Washington, Seattle *Google, Seattle {mausam,soderlan,etzioni,weld,bilmes}@cs.washington.edu mskinner@google.com Abstract Can we automatically compose a large set of Wiktionaries and translation dictionar- ies to yield a massive, multilingual dic- tionary whose coverage is substantially greater than that of any of its constituent dictionaries? The composition of multiple translation dictionaries leads to a transitive inference problem: if word A translates to word B which in turn translates to word C, what is the probability that C is a trans- lation of A? The paper introduces a novel algorithm that solves this problem for 10,000,000 words in more than 1,000 languages. The algorithm yields PANDIC- TIONARY, a novel multilingual dictionary. PANDICTIONARY contains more than four times as many translations than in the largest Wiktionary at precision 0.90 and over 200,000,000 pairwise translations in over 200,000 language pairs at precision 0.8. 1 Introduction and Motivation In the era of globalization, inter-lingual com- munication is becoming increasingly important. Although nearly 7,000 languages are in use to- day (Gordon, 2005), most language resources are mono-lingual, or bi-lingual.1 This paper investi- gates whether Wiktionaries and other translation dictionaries available over the Web can be auto- matically composed to yield a massive, multilin- gual dictionary with superior coverage at compa- rable precision. We describe the automatic construction of a massive multilingual translation dictionary, called 1The English Wiktionary, a lexical resource developed by volunteers over the Internet is one notable exception that con- tains translations of English words in about 500 languages. Figure 1: A fragment of the translation graph for two senses of the English word ‘spring’. Edges labeled ‘1’ and ‘3’ are for spring in the sense of a season, and ‘2’ and ‘4’ are for the flexible coil sense. The graph shows translation entries from an English dictionary merged with ones from a French dictionary. PANDICTIONARY, that could serve as a resource for translation systems operating over a very broad set of language pairs. The most immedi- ate application of PANDICTIONARY is to lexical translation—the translation of individual words or simple phrases (e.g., “sweet potato”). Because lexical translation does not require aligned cor- pora as input, it is feasible for a much broader set of languages than statistical Machine Transla- tion (SMT). Of course, lexical translation cannot replace SMT, but it is useful for several applica- tions including translating search-engine queries, library classifications, meta-data tags,2 and recent applications like cross-lingual image search (Et- zioni et al., 2007), and enhancing multi-lingual Wikipedias (Adar et al., 2009). Furthermore, lexical translation is a valuable component in knowledge-based Machine Translation systems, e.g., (Bond et al., 2005; Carbonell et al., 2006). PANDICTIONARY currently contains over 200 million pairwise translations in over 200,000 lan- guage pairs at precision 0.8. It is constructed from information harvested …

Content truncated to 500 words