Distributed Minimum Error Rate Training of SMT using Particle Swarm Optimization

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 649–657, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Distributed Minimum Error Rate Training of SMT using Particle Swarm Optimization Jun Suzuki, Kevin Duh, and Masaaki Nagata NTT Communication Science Laboratories, NTT Corp. 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237 Japan {suzuki.jun, kevin.duh, nagata.masaaki}@lab.ntt.co.jp Abstract The direct optimization of a translation metric is an integral part of building state- of-the-art SMT systems. Unfortunately, widely used translation metrics such as BLEU-score are non-smooth, non-convex, and non-trivial to optimize. Thus, stan- dard optimizers such as minimum error rate training (MERT) can be extremely time-consuming, leading to a slow turn- around rate for SMT research and exper- imentation. We propose an alternative ap- proach based on particle swarm optimiza- tion (PSO), which can easily exploit the fast growth of distributed computing to obtain solutions quickly. For example in our experiments on NIST 2008 Chinese- to-English data with 512 cores, we demon- strate a speed increase of up to 15x and reduce the parameter tuning time from 10 hours to 40 minutes with no degradation in BLEU-score. 1 Introduction Recent statistical machine translation (SMT) sys- tems employ a linear combination of several model components, such as translation models, language models, and reordering models. Trans- lation is performed by selecting the most-likely translation, which is the candidate translation with the highest score based on the different model components. We usually refer to this search pro- cess as ‘decoding’. Although the development of decoding algorithms is a key topic in SMT re- search, if we are to construct better SMT systems it is also important to find a way to determine the weights of different model components. We refer to this process as ‘parameter tuning’. The current standard strategy for tuning the pa- rameters of SMT systems is to search for the weights that maximize a given translation quality metric such as BLEU-score (Papineni et al., 2002). In fact, minimum error rate training (MERT) pro- posed by (Och, 2003) is the most widely used pa- rameter tuning method in SMT community. This is because, empirically, we obtain better trans- lation system performance by directly optimiz- ing the translation metric than by maximizing the likelihood function. However, translation met- rics such as BLEU-score are often non-smooth, non-convex, and non-trivial to optimize, and di- rect maximization does not allow us simply to ap- ply well-known fast and robust optimization tech- niques such as the gradient-ascent method. This restriction makes the parameter tuning process ex- tremely slow. In this paper, we propose a novel distributed MERT framework based on particle swarm opti- mization (PSO) (Kennedy and Eberhart, 1995), a population based stochastic optimization tech- nique, to improve the slow parameter tuning pro- cess. The main characteristics of our method are that: (1) it directly optimizes a metric such as BLEU-score, (2) it greatly reduces the parame- ter tuning time compared with the conventional MERT, and (3) it is highly scalable, meaning that additional distributed processors lead …

Content truncated to 500 words