Top-Rank Enhanced Listwise Optimization for Statistical Machine Translation

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 90–99, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Top-Rank Enhanced Listwise Optimization for Statistical Machine Translation Huadong Chen,† Shujian Huang,†∗ David Chiang,‡ Xinyu Dai,† Jiajun Chen† †State Key Laboratory for Novel Software Technology, Nanjing University {chenhd,huangsj,daixinyu,chenjj}@nlp.nju.edu.cn ‡Department of Computer Science and Engineering, University of Notre Dame dchiang@nd.edu Abstract Pairwise ranking methods are the basis of many widely used discriminative train- ing approaches for structure prediction problems in natural language processing (NLP). Decomposing the problem of rank- ing hypotheses into pairwise comparisons enables simple and efficient solutions. However, neglecting the global ordering of the hypothesis list may hinder learning. We propose a listwise learning framework for structure prediction problems such as machine translation. Our framework di- rectly models the entire translation list’s ordering to learn parameters which may better fit the given listwise samples. Fur- thermore, we propose top-rank enhanced loss functions, which are more sensitive to ranking errors at higher positions. Exper- iments on a large-scale Chinese-English translation task show that both our list- wise learning framework and top-rank en- hanced listwise losses lead to significant improvements in translation quality. 1 Introduction Discriminative training methods for structured prediction in natural language processing (NLP) aim to estimate the parameters of a model that as- signs a score to each hypothesis in the (possibly very large) search space. For example, in statisti- cal machine translation (SMT), the model assigns a score to each possible translation, and in syn- tactic parsing, the function assigns a score to each possible syntactic tree. Ideally, the model should assign scores that rank hypotheses according to their true quality. In this paper, we consider the problem of discriminative training for SMT. ∗Corresponding author. Traditional SMT systems use log-linear models with only about a dozen features, such as trans- lation probabilities and language model probabil- ities (Yamada and Knight, 2001; Koehn et al., 2003; Chiang, 2005; Liu et al., 2006). These mod- els can be tuned by minimum error rate training (MERT) (Och, 2003), which directly optimizes BLEU using coordinate ascent combined with a global line search. To enable training of modern SMT systems, which can have thousands of features or more, many research efforts have been made towards scalable discriminative training methods (Chiang et al., 2008; Hopkins and May, 2011; Bazrafshan et al., 2012). Most of these methods either de- fine loss functions that push the model to correctly compare pairs of hypotheses, or use approximate optimization methods that effectively do the same. For practical reasons, only a subset of the pairs are considered; these pairs are selected by either sam- pling (Hopkins and May, 2011) or heuristic meth- ods (Watanabe et al., 2007; Chiang et al., 2008). But this pairwise approach neglects the global ordering of the list of hypotheses, which may lead to problems trying to learn good parameter values. Inspired by research in information re- trieval (IR) (Cao et al., 2007; Xia et al., 2008), …

Content truncated to 500 words