Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length

Proceedings of the 19th Conference on Computational Language Learning, pages 62–72, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length Francisco Guzm´an Preslav Nakov ALT Research Groupand Stephan Vogel Qatar Computing Research Institute, HBKU {fguzman,pnakov,svogel}@qf.org.qa Abstract We study the impact of source length and verbosity of the tuning dataset on the per- formance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensi- tive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high- verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reorder- ing, number of unknown words, syntac- tic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimiza- tion strategy does not overshadow scien- tific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend us- ing PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs. 1 Introduction Statistical machine translation (SMT) systems nowadays are complex and consist of many com- ponents such as a translation model, a reorder- ing model, a language model, etc., each of which could have several sub-components. All compo- nents and their elements work together to score full and partial hypotheses proposed by the SMT system’s search algorithms. Thus, putting them together requires assigning them relative weights, e.g., how much weight we should give to the translation model vs. the lan- guage model vs. the reordering table. These rela- tive weights are typically learned discriminatively in a log-linear framework, and their values are op- timized to maximize some automatic metric, typi- cally BLEU, on a tuning dataset. Given this setup, it is clear that the choice of a tuning set and its characteristics, can have signif- icant impact on the SMT system’s performance: if the experimental framework (training data, tun- ing set, and test set) is highly consistent, i.e., there is close similarity in terms of genre, domain and verbosity,1 then translation quality can be im- proved by careful selection of tuning sentences that exhibit high degree of similarity to the test set (Zheng et al., 2010; Li et al., 2010). In our recent work (Nakov et al., 2012), we have studied the relationship between optimizers such as MERT, PRO and MIRA, and we have pointed out that PRO tends to generate relatively shorter translations, which could lead to lower BLEU scores on testing. Our solution there was to fix the objective function being optimized: PRO uses sentence-level smoothed BLEU+1, as opposed to the standard dataset-level …

Content truncated to 500 words