Rare Word Translation Extraction from Aligned Comparable Documents

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1327–1335, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Rare Word Translation Extraction from Aligned Comparable Documents Emmanuel Prochasson and Pascale Fung Human Language Technology Center Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong {eemmanuel,pascale}@ust.hk Abstract We present a first known result of high pre- cision rare word bilingual extraction from comparable corpora, using aligned compara- ble documents and supervised classification. We incorporate two features, a context-vector similarity and a co-occurrence model between words in aligned documents in a machine learning approach. We test our hypothesis on different pairs of languages and corpora. We obtain very high F-Measure between 80% and 98% for recognizing and extracting cor- rect translations for rare terms (from 1 to 5 oc- currences). Moreover, we show that our sys- tem can be trained on a pair of languages and test on a different pair of languages, obtain- ing a F-Measure of 77% for the classification of Chinese-English translations using a train- ing corpus of Spanish-French. Our method is therefore even potentially applicable to low re- sources languages without training data. 1 Introduction Rare words have long been a challenge to translate automatically using statistical methods due to their low occurrences. However, the Zipf’s Law claims that, for any corpus of natural language text, the fre- quency of a word wn (n being its rank in the fre- quency table) will be roughly twice as high as the frequency of word wn+1. The logical consequence is that in any corpus, there are very few frequent words and many rare words. We propose a novel approach to extract rare word translations from comparable corpora, relying on two main features. The first feature is the context-vector similar- ity (Fung, 2000; Chiao and Zweigenbaum, 2002; Laroche and Langlais, 2010): each word is charac- terized by its context in both source and target cor- pora, words in translation should have similar con- text in both languages. The second feature follows the assumption that specific terms and their translations should appear together often in documents on the same topic, and rarely in non-related documents. This is the gen- eral assumption behind early work on bilingual lex- icon extraction from parallel documents using sen- tence boundary as the context window size for co- occurrence computation, we suggest to extend it to aligned comparable documents using document as the context window. This document context is too large for co-occurrence computation of functional words or high frequency content words, but we show through observations and experiments that this win- dow size is appropriate for rare words. Both these features are unreliable when the num- ber of occurrences of words are low. We sug- gest however that they are complementary and can be used together in a machine learning approach. Moreover, we suggest that the model trained for one pair of languages can be successfully applied to ex- tract translations from another pair of languages. …

Content truncated to 500 words