Online Large-Margin Training for Statistical Machine Translation

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 764–773, Prague, June 2007. c⃝2007 Association for Computational Linguistics Online Large-Margin Training for Statistical Machine Translation Taro Watanabe Jun Suzuki Hajime Tsukada NTT Communication Science LaboratoriesHideki Isozaki 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237 Japan {taro,jun,tsukada,isozaki}@cslab.kecl.ntt.co.jp Abstract We achieved a state of the art performance in statistical machine translation by using a large number of features with an online large-margin training algorithm. The mil- lions of parameters were tuned only on a small development set consisting of less than 1K sentences. Experiments on Arabic-to- English translation indicated that a model trained with sparse binary features outper- formed a conventional SMT system with a small number of features. 1 Introduction The recent advances in statistical machine transla- tion have been achieved by discriminatively train- ing a small number of real-valued features based ei- ther on (hierarchical) phrase-based translation (Och and Ney, 2004; Koehn et al., 2003; Chiang, 2005) or syntax-based translation (Galley et al., 2006). How- ever, it does not scale well with a large number of features of the order of millions. Tillmann and Zhang (2006), Liang et al. (2006) and Bangalore et al. (2006) introduced sparse binary features for statistical machine translation trained on a large training corpus. In this framework, the prob- lem of translation is regarded as a sequential labeling problem, in the same way as part-of-speech tagging, chunking or shallow parsing. However, the use of a large number of features did not provide any signifi- cant improvements over a conventional small feature set. Bangalore et al. (2006) trained the lexical choice model by using Conditional Random Fields (CRF) realized on a WFST. Their modeling was reduced to Maximum Entropy Markov Model (MEMM) to han- dle a large number of features which, in turn, faced the labeling bias problem (Lafferty et al., 2001). Tillmann and Zhang (2006) trained their feature set using an online discriminative algorithm. Since the decoding is still expensive, their online training ap- proach is approximated by enlarging a merged k- best list one-by-one with a 1-best output. Liang et al. (2006) introduced an averaged perceptron al- gorithm, but employed only 1-best translation. In Watanabe et al. (2006a), binary features were trained only on a small development set using a variant of voted perceptron for reranking k-best translations. Thus, the improvement is merely relative to the baseline translation system, namely whether or not there is a good translation in their k-best. We present a method to estimate a large num- ber of parameters — of the order of millions — using an online training algorithm. Although it was intuitively considered to be prone to overfit- ting, training on a small development set — less than 1K sentences — was sufficient to achieve im- proved performance. In this method, each train- ing sentence is decoded and weights are updated at every iteration (Liang et al., 2006). When updat- ing model parameters, we employ a memorization- variant of …

Content truncated to 500 words