Why Neural Translations are the Right Length

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2278–2282, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Why Neural Translations are the Right Length Xing Shi1, Kevin Knight1, and Deniz Yuret2 1Information Sciences Institute & Computer Science Department University of Southern California {xingshi,knight}@isi.edu 2Computer Engineering, Koc¸ University dyuret@ku.edu.tr Abstract We investigate how neural, encoder-decoder translation systems output target strings of ap- propriate lengths, finding that a collection of hidden units learns to explicitly implement this functionality. 1 Introduction The neural encoder-decoder framework for machine translation (Neco and Forcada, 1997; Casta˜no and Casacuberta, 1997; Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015) provides new tools for addressing the field’s difficult challenges. In this framework (Figure 1), we use a recurrent neural net- work (encoder) to convert a source sentence into a dense, fixed-length vector. We then use another re- current network (decoder) to convert that vector into a target sentence. In this paper, we train long short- term memory (LSTM) neural units (Hochreiter and Schmidhuber, 1997) trained with back-propagation through time (Werbos, 1990). A remarkable feature of this simple neural MT (NMT) model is that it produces translations of the right length. When we evaluate the system on previ- ously unseen test data, using BLEU (Papineni et al., 2002), we consistently find the length ratio between MT outputs and human references translations to be very close to 1.0. Thus, no brevity penalty is in- curred. This behavior seems to come for free, with- out special design. By contrast, builders of standard statistical MT (SMT) systems must work hard to ensure correct length. The original mechanism comes from the IBM SMT group, whose famous Models 1-5 in- cluded a learned table ϵ(y|x), with x and y being the lengths of source and target sentences (Brown et al., 1993). But they did not deploy this table when decoding a foreign sentence f into an English sen- tence e; it did not participate in incremental scoring and pruning of candidate translations. As a result (Brown et al., 1995): “However, for a given f, if the goal is to discover the most probable e, then the product P(e) P(f|e) is too small for long English strings as compared with short ones. As a result, short English strings are im- properly favored over longer English strings. This tendency is counteracted in part by the following modification: Replace P(f|e) with clength(e) · P(f|e) for some empirically chosen constant c. This modifi- cation is treatment of the symptom rather than treat- ment of the disease itself, but it offers some tempo- rary relief. The cure lies in better modeling.” More temporary relief came from Minimum Error-Rate Training (MERT) (Och, 2003), which au- tomatically sets c to maximize BLEU score. MERT also sets weights for the language model P(e), trans- lation model P(f|e), and other features. The length feature combines so sensitively with other features that MERT frequently returns to it as it revises one weight at a …

Content truncated to 500 words