Expected BLEU Training for Graphs: BBN System Description for WMT11 System Combination Task

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 159–165, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Expected BLEU Training for Graphs: BBN System Description for WMT11 System Combination Task Antti-Veikko I. Rosti∗and Bing Zhang and Spyros Matsoukas and Richard Schwartz Raytheon BBN Technologies, 10 Moulton Street, Cambridge, MA 02138, USA {arosti,bzhang,smatsouk,schwartz}@bbn.com Abstract BBN submitted system combination outputs for Czech-English, German-English, Spanish- English, and French-English language pairs. All combinations were based on confusion network decoding. The confusion networks were built using incremental hypothesis align- ment algorithm with flexible matching. novel bi-gram count feature, which can penal-A ize bi-grams not present in the input hypothe- ses corresponding to a source sentence, was introduced in addition to the usual decoder features. The system combination weights were tuned using a graph based expected BLEU as the objective function while incre- mentally expanding the networks to bi-gram and 5-gram contexts. The expected BLEU tuning described in this paper naturally gen- eralizes to hypergraphs and can be used to optimize thousands of weights. The com- bination gained about 0.5-4.0 BLEU points over the best individual systems on the official WMT11 language pairs. A 39 system multi- source combination achieved an 11.1 BLEU point gain. 1 Introduction The confusion networks for the BBN submissions to the WMT11 system combination task were built using incremental hypothesis alignment algorithm ∗This work was supported by DARPA/I2O Contract No. HR0011-06-C-0022 under the GALE program (Approved for Public Release, Distribution Unlimited). The views, opinions, and/or findings contained in this article/presentation are those of the author/presenter and should not be interpreted as represent- ing the official views or policies, either expressed or implied, of the Defense Advanced Research Projects Agency or the De- partment of Defense. with flexible matching (Rosti et al., 2009). A novel bi-gram count feature was used in addition to the standard decoder features. The N-best list based ex- pected BLEU tuning (Rosti et al., 2010), similar to the one proposed by Smith and Eisner (2006), was extended to operate on word lattices. This method is closely related to the consensus BLEU (CoBLEU) proposed by Pauls et al. (2009). The minimum oper- ation used to compute the clipped counts (matches) in the BLEU score (Papineni et al., 2002) was re- placed by a differentiable function, so there was no need to use sub-gradient ascent as in CoBLEU. The expected BLEU (xBLEU) naturally generalizes to hypergraphs by simply replacing the forward- backward algorithm with inside-outside algorithm when computing the expected n-gram counts and sufficient statistics for the gradient. The gradient ascent optimization of the xBLEU appears to be more stable than the gradient-free di- rect 1-best BLEU tuning or N-best list based min- imum error rate training (Och, 2003), especially when tuning a large number of weights. On the of- ficial WMT11 language pairs with up to 30 weights, there was no significant benefit from maximizing xBLEU. However, on a 39 system multi-source combination (43 weights total), it yielded a signif- icant gain over gradient-free …

Content truncated to 500 words