Speed-Constrained Tuning for Statistical Machine Translation Using Bayesian Optimization

Proceedings of NAACL-HLT 2016, pages 856–865, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Speed-Constrained Tuning for Statistical Machine Translation Using Bayesian Optimization Daniel Beck†∗ Adri`a de Gispert‡ Gonzalo Iglesias‡ Aurelien Waite‡ Bill Byrne‡ †Department of Computer Science, University of Sheffield, United Kingdom debeck1@sheffield.ac.uk ‡SDL Research, Cambridge, United Kingdom {agispert,giglesias,rwaite,bbyrne}@sdl.com Abstract We address the problem of automatically finding the parameters of a statistical ma- chine translation system that maximize BLEU scores while ensuring that decoding speed ex- ceeds a minimum value. We propose the use of Bayesian Optimization to efficiently tune the speed-related decoding parameters by eas- ily incorporating speed as a noisy constraint function. The obtained parameter values are guaranteed to satisfy the speed constraint with an associated confidence margin. Across three language pairs and two speed constraint val- ues, we report overall optimization time re- duction compared to grid and random search. We also show that Bayesian Optimization can decouple speed and BLEU measurements, re- sulting in a further reduction of overall op- timization time as speed is measured over a small subset of sentences. 1 Introduction Research in Statistical Machine Translation (SMT) aims to improve translation quality, typically mea- sured by BLEU scores (Papineni et al., 2001), over a baseline system. Given a task defined by a lan- guage pair and its corpora, the quality of a system is assessed by contrasting choices made in rule/phrase extraction criteria, feature functions, decoding algo- rithms and parameter optimization techniques. Some of these choices result in systems with sig- nificant differences in performance. For example, in phrase-based translation (PBMT) (Koehn et al., ∗ This work was done during an internship of the first au- thor at SDL Research, Cambridge. 2003), decoder parameters such as pruning thresh- olds and reordering constraints can have a dramatic impact on both BLEU and decoding speed. How- ever, unlike feature weights, which can be optimized by MERT (Och and Ney, 2004), it is difficult to op- timize decoder parameters either for speed or for BLEU. We are interested in the problem of automatically finding the decoder parameters and feature weights that yield the best BLEU at a specified minimum de- coding speed. This is potentially very expensive be- cause each change in a decoder parameter requires re-decoding to assess both BLEU and translation speed. This is under-studied in the literature, despite its importance for real-life commercial SMT engines whose speed and latency can be as significant for user satisfaction as overall translation quality. We propose to use Bayesian Optimiza- tion (Brochu et al., 2010b; Shahriari et al., 2015) for this constrained optimization task. By using prior knowledge of the function to be optimized and by exploring the most uncertain and the most promising regions of the parameter space, Bayesian Optimization (BO) is able to quickly find optimal parameter values. It is particularly well-suited to optimize expensive and non-differentiable functions such as the BLEU score of a decoder on a tuning set. The BO framework can also incorporate noisy constraints, such as decoder …

Content truncated to 500 words