Using Similarity Scoring to Improve the Bilingual Dictionary for Sub-sentential Alignment

Using Similarity Scoring To Improve the Bilingual Dictionary for Word Alignment Katharina Probst Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA, 15213 kathrin@cs.cmu.edu Ralf Brown Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA, 15213 ralf@cs.cmu.edu Abstract We describe an approach to improve the bilingual cooccurrence dictionary that is used for word alignment, and evaluate the improved dictionary using a version of the Competitive Linking algorithm. We demonstrate a problem faced by the Com- petitive Linking algorithm and present an approach to ameliorate it. In particular, we rebuild the bilingual dictionary by cluster- ing similar words in a language and as- signing them a higher cooccurrence score with a given word in the other language than each single word would have other- wise. Experimental results show a signifi- cant improvement in precision and recall for word alignment when the improved dicitonary is used. 1 Introduction and Related Work Word alignment is a well-studied problem in Natu- ral Language Computing. This is hardly surprising given its significance in many applications: word- aligned data is crucial for example-based machine translation, statistical machine translation, but also other applications such as cross-lingual information retrieval. Since it is a hard and time-consuming task to hand-align bilingual data, the automation of this task receives a fair amount of attention. In this pa- per, we present an approach to improve the bilin- gual dictionary that is used by word alignment al- gorithms. Our method is based on similarity scores between words, which in effect results in the clus- tering of morphological variants. One line of related work is research in clustering based on word similarities. This problem is an area of active research in the Information Retrieval com- munity. For instance, Xu and Croft (1998) present an algorithm that first clusters what are assumedly variants of the same word, then further refines the clusters using a cooccurrence related measure. Word variants are found via a stemmer or by clustering all words that begin with the same three letters. An- other technique uses similarity scores based on N- grams (e.g. (Kosinov, 2001)). The similarity of two words is measured using the number of N-grams that their occurrences have in common. As in our ap- proach, similar words are then clustered into equiv- alence classes. Other related work falls in the category of word alignment, where much research has been done. A number of algorithms have been proposed and eval- uated for the task. As Melamed (2000) points out, most of these algorithms are based on word cooccur- rences in sentence-aligned bilingual data. A source language word �� and a target language word � � are said to cooccur if �� occurs in a source language sen- tence and � � occurs in the corresponding target lan- guage sentence. Cooccurrence scores then are then counts for all word pairs �� and �� , where �� is in the source language vocabulary and ��� is in the tar- get language vocabulary. Often, the scores also take into account the …

Content truncated to 500 words