Corpus Variations for Translation Lexicon Induction

Corpus Variations for Translation Lexicon Induction Rebecca Hwa∗ Carol Nichols§ Khalil Sima’an¶ ∗Department of Computer Science University of Pittsburgh hwa@cs.pitt.edu § Vivisimo Inc. carol.nichols@gmail.com ¶ FNWI University of Amsterdam simaan@science.uva.nl Abstract Lexical mappings (word translations) between languages are an invaluable resource for mul- tilingual processing. While the problem of extracting lexical mappings from parallel cor- pora is well-studied, the task is more challeng- ing when the language samples are from non- parallel corpora. The goal of this work is to investigate one such scenario: finding lexical mappings between dialects of a diglossic lan- guage, in which people conduct their written communications in a prestigious formal dialect, but they communicate verbally in a colloquial dialect. Because the two dialects serve dif- ferent socio-linguistic functions, parallel cor- pora do not naturally exist between them. An example of a diglossic dialect pair is Modern Standard Arabic (MSA) and Levantine Ara- bic. In this paper, we evaluate the applicabil- ity of a standard algorithm for inducing lexical mappings between comparable corpora (Rapp, 1999) to such diglossic corpora pairs. The fo- cus of the paper is an in-depth error analysis, exploring the notion of relatedness in diglossic corpora and scrutinizing the effects of various dimensions of relatedness (such as mode, topic, style, and statistics) on the quality of the result- ing translation lexicon. 1 Introduction A translation lexicon is an important component of mul- tilingual processing applications such as machine transla- tion systems (Brown et al., 1990; Al-Onaizan et al., 1999) and multilingual information retrieval systems (Sheridan and Ballerini, 1996; CLE, 2005). A translation lexi- con can also facilitate cross-lingual resource building. For example, Yarowsky and Ngai (2001) have shown that, for training a French part-of-speech tagger, one could acquire an automatically annotated training cor- pus by projecting it from the output of an English part- of-speech tagger via lexical translations between English and French. Abstractly speaking, a translation lexicon is a mapping between two disjoint sets of symbols. Given some corpus sample over each set of symbols, one might induce the mapping by performing statistical analyses on the cor- pora to find correlations between the symbols. The qual- ity of the mapping depends on the degree of relatedness between the corpora. Parallel corpora, in which every pair of sentences is a translation of each other, facilitate the induction of a mapping between word tokens (situated occurrences); in contrast, one might only be able to glean a mapping between word types (as in a wide coverage dictionary) from non-parallel corpora. The induction of mappings between word tokens from parallel corpora has been extensively studied; there ex- ist many alignment methods, both supervised and un- supervised, that yield highly accurate lexical mappings between word tokens (Melamed, 2000; Och and Ney, 2003; Callison-Burch et al., 2004). However, parallel corpora are not always available. For instance, consider the problem of finding a mapping between two dialects of a diglossic language (i.e., the language exists in two forms: a “prestigious” variety for formal communications and a colloquial variety for …

Content truncated to 500 words