Low Cost Portability for Statistical Machine Translation based on N-gram Frequency and TF-IDF

Low Cost Portability for Statistical Machine Translation based on N-gram Frequency and TF-IDF Matthias Eck, Stephan Vogel and Alex Waibel Interactive Systems Laboratories Carnegie Mellon University Pittsburgh, PA, 15213, USA matteck@cs.cmu.edu, vogel+@cs.cmu.edu, waibel@cs.cmu.edu Abstract Statistical machine translation relies heavily on the available training data. In some cases it is necessary to limit the amount of training data that can be created for or actually used by the systems. We introduce weighting schemes which allow us to sort sentences based on the frequency of unseen n-grams. A second approach uses TF-IDF to rank the sentences. After sorting we can select smaller training corpora and we are able to show that systems trained on much less training data achieve a very competitive performance compared to baseline systems using all available training data. 1. Introduction The goal of this research was to decrease the amount of training data that is necessary to train a competitive statistical translation system regardless of the actual test data or its domain. “Competitive” here means that the system should not produce significantly worse translations compared to a system trained on a significantly larger amount of data. It is important to note that this is not an adaptation approach as we assume that the test data (and its domain) is not known at the time we select the actual training data. Statistical machine translation can be described in a formal way as follows: () (|) argmax (|) argmax * Pt Pst Pts t t t ⋅ = = Here t is the target sentence, and s is the source sentence. P(t) is the target language model and P(s|t) is the translation model used in the decoder. Statistical machine translation searches for the best target sentence from the space defined by the target language model and the translation model. Statistical translation models are usually either phrase- or word-based and include most notably IBM1 to IBM4 and HMM ([1], [2], [3]). Some recent developments focused on online phrase extraction ([4], [5]). All models use available bilingual training data in the source and target languages to estimate their parameters and approximate the translation probabilities. One of the main problems of Statistical Machine Translation (SMT) is the necessity to have large parallel corpora available. This might not be a big issue for major languages, but it certainly is a problem for languages with fewer resources ([6], [7]). To improve the data situation for these languages it is necessary to hire human translators at enormous costs who translate corpora that can later be used to train SMT systems. Our idea focuses on sorting the available source sentences that should be translated by a human translator according to their approximate importance. The importance is estimated using a frequency based and an information retrieval approach. 2. Motivation There are three inherently different motivations for the goal of limiting the amount of necessary training data for a competitive translation system. We described those motivations and their applications already in the paper [8]. Application 1: Reducing Human Translation …

Content truncated to 500 words