Phrase Pair Rescoring with Term Weighting for Statistical Machine Translation

Phrase Pair Rescoring with Term Weightings for Statistical Machine Translation Bing Zhao Stephan Vogel Alex Waibel Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA, 15213, USA {bzhao, vogel+, ahw}@cs.cmu.edu Abstract We propose to score phrase translation pairs for statistical machine translation using term weight based models. These models employ tf.idf to encode the weights of content and non-content words in phrase translation pairs. The transla- tion probability is then modeled by simi- larity functions defined in a vector space. Two similarity functions are compared. Using these models in a statistical ma- chine translation task shows significant improvements. 1 Introduction Words can be classified as content and func- tional words. Content words like verbs and proper nouns are more informative than function words like "to'' and "the''. In machine translation, intuitively, the informative content words should be emphasized more for better adequacy of the translation quality. However, the standard statis- tical translation approach does not take account how informative and thereby, how important a word is, in its translation model. One reason is the difficulty to measure how informative a word is. Another problem is to integrate it naturally into the existing statistical machine translation framework, which typically is built on word alignment models, like the well-known IBM alignment models (Brown et al 1993). In recent years there has been a strong ten- dency to incorporate phrasal translation into sta- tistical machine translation. It directly translates an n-gram from the source language into an m- gram in the target language. The advantages are obvious: It has built-in local context modeling, and provides reliable local word reordering. It has multi-word translations, and models a word’s conditional fertility given a local context. It cap- tures idiomatic phrase translations and can be easily enriched with bilingual dictionaries. In addition, it can compensate for the segmentation errors made during preprocessing, i.e. word seg- mentation errors of Chinese. The advantage of using phrase-based translation in a statistical framework has been shown in many studies such as (Koehn et al. 2003; Vogel et al. 2003; Zens et al. 2002; Marcu and Wong, 2002). However, the phrase translation pairs are typically extracted from a parallel corpus based on the Viterbi align- ment of some word alignment models. The leads to the question what probability should be as- signed to those phrase translations. Different approaches have been suggested as using relative frequencies (Zens et al. 2002), calculate prob- abilities based on a statistical word-to-word dic- tionary (Vogel et al. 2003) or use a linear interpolation of these scores (Koehn et al. 2003). In this paper we investigate a different ap- proach with takes the information content of words better into account. Term weighting based vector models are proposed to encode the transla- tion quality. The advantage is that term weights, such as tf.idf, are useful to model the informa- tiveness of words. Highly informative content words usually have high tf.idf scores. In informa- tion retrieval this has been successfully applied to capture the relevance of a document to a …

Content truncated to 500 words