Robust Measurement and Comparison of Context Similarity for Finding Translation Pairs

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 19–27, Beijing, August 2010 Robust Measurement and Comparison of Context Similarity for Finding Translation Pairs Daniel Andrade†, Tetsuya Nasukawa‡, Jun’ichi Tsujii† †Department of Computer Science, University of Tokyo {daniel.andrade, tsujii}@is.s.u-tokyo.ac.jp ‡IBM Research - Tokyo nasukawa@jp.ibm.com Abstract In cross-language information retrieval it is often important to align words that are similar in meaning in two corpora writ- ten in different languages. Previous re- search shows that using context similar- ity to align words is helpful when no dictionary entry is available. We sug- gest a new method which selects a sub- set of words (pivot words) associated with a query and then matches these words across languages. To detect word associa- tions, we demonstrate that a new Bayesian method for estimating Point-wise Mutual Information provides improved accuracy. In the second step, matching is done in a novel way that calculates the chance of an accidental overlap of pivot words us- ing the hypergeometric distribution. We implemented a wide variety of previously suggested methods. Testing in two con- ditions, a small comparable corpora pair and a large but unrelated corpora pair, both written in disparate languages, we show that our approach consistently out- performs the other systems. 1 Introduction Translating domain-specific, technical terms from one language to another can be challenging be- cause they are often not listed in a general dictio- nary. The problem is exemplified in cross-lingual information retrieval (Chiao and Zweigenbaum, 2002) restricted to a certain domain. In this case, the user might enter only a few technical terms. However, jargons that appear frequently in the data set but not in general dictionaries, impair the usefulness of such systems. Therefore, various means to extract translation pairs automatically have been proposed. They use different clues, mainly • Spelling distance or transliterations, which are useful to identify loan words (Koehn and Knight, 2002). • Context similarity, helpful since two words with identical meaning are often used in sim- ilar contexts across languages (Rapp, 1999). The first type of information is quite specific; it can only be helpful in a few cases, and can thereby engender high-precision systems with low recall, as described for example in (Koehn and Knight, 2002). The latter is more general. It holds for most words including loan words. Usually the context of a word is defined by the words which occur around it (bag-of-words model). Let us briefly recall the main idea for using context similarity to find translation pairs. First, the degree of association between the query word and all content words is measured with respect to the corpus at hand. The same is done for every possible translation candidate in the target cor- pus. This way, we can create a feature vector for the query and all its possible translation can- didates. We can assume that, for some content words, we have valid translations in a general dic- tionary, which enables us to compare the vectors across languages. We will designate these content …

Content truncated to 500 words