A Discriminative Training Procedure for Continuous Translation Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1046–1052, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Discriminative Training Procedure for Continuous Translation Models † ∗Quoc-Khanh Do, † ∗Alexandre Allauzen and ∗Franc¸ois Yvon † Universit´e Paris-Sud, Orsay, France ∗ LIMSI/CNRS, Orsay, France firstname.surname@limsi.fr Abstract Continuous-space translation models have recently emerged as extremely powerful ways to boost the performance of existing translation systems. A simple, yet effec- tive way to integrate such models in infer- ence is to use them in an N-best rescor- ing step. In this paper, we focus on this scenario and show that the performance gains in rescoring can be greatly increased when the neural network is trained jointly with all the other model parameters, using an appropriate objective function. Our ap- proach is validated on two domains, where it outperforms strong baselines. 1 Introduction Over the past few years, research on neural net- works (NN) architectures for Natural Language Processing has been rejuvenated. Boosted by early successes in language modelling for speech recog- nition (Schwenk, 2007; Le et al., 2011), NNs have since been successufully applied to many other tasks (Socher et al., 2013; Huang et al., 2012; Yang et al., 2013). In particular, these techniques have been applied to Statistical Machine Trans- lation (SMT), first to estimate continuous-space translation models (CTMs) (Schwenk et al., 2007; Le et al., 2012; Devlin et al., 2014), and more recently to implement end-to-end translation sys- tems (Cho et al., 2014; Sutskever et al., 2014). In most SMT settings, CTMs are used as an ad- ditional feature function in the log-linear model, and are conventionally trained by maximizing the regularized log-likelihood on some parallel train- ing corpora. Since this objective function requires to normalize scores, several alternative training objectives have recently been proposed to speed up training and inference, a popular and effec- tive choice being the Noise Contrastive Estimation (NCE) introduced in (Gutmann and Hyv¨arinen, 2010). In any case, NN training is typically per- formed (a) in isolation from the other components of the SMT system and (b) using a criterion that is unrelated to the actual performance of the SMT system (as measured for instance by BLEU). It is therefore likely that the resulting NN parameters are sub-optimal with respect to their intended use. In this paper, we study an alternative training regime aimed at addressing problems (a) and (b). To this end, we propose a new objective func- tion used to discriminatively train or adapt CTMs, along with a training procedure that enables to take the other components of the system into account. Our starting point is a non-normalized extension of the n-gram CTM of (Le et al., 2012) that we briefly restate in section 2. We then introduce our objective function and the associated optimization procedure in section 3. As will be discussed, our new training criterion is inspired both from max- margin methods (Watanabe et al., 2007) and from pair-wise ranking (PRO) (Hopkins and May, 2011; Simianer …

Content truncated to 500 words