Unfolding and Shrinking Neural Machine Translation Ensembles

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1946–1956 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Unfolding and Shrinking Neural Machine Translation Ensembles Felix Stahlberg and Bill Byrne {fs439,wjb31}@cam.ac.uk Department of Engineering University of Cambridge, UK Abstract Ensembling is a well-known technique in neural machine translation (NMT) to im- prove system performance. Instead of a single neural net, multiple neural nets with the same topology are trained separately, and the decoder generates predictions by averaging over the individual models. En- sembling often improves the quality of the generated translations drastically. How- ever, it is not suitable for production sys- tems because it is cumbersome and slow. This work aims to reduce the runtime to be on par with a single system without com- promising the translation quality. First, we show that the ensemble can be unfolded into a single large neural network which imitates the output of the ensemble sys- tem. We show that unfolding can already improve the runtime in practice since more work can be done on the GPU. We pro- ceed by describing a set of techniques to shrink the unfolded network by reducing the dimensionality of layers. On Japanese- English we report that the resulting net- work has the size and decoding speed of a single NMT network but performs on the level of a 3-ensemble system. 1 Introduction The top systems in recent machine translation evaluation campaigns on various language pairs use ensembles of a number of NMT systems (Bo- jar et al., 2016; Sennrich et al., 2016a; Chung et al., 2016; Neubig, 2016; Wu et al., 2016; Cromieres et al., 2016; Durrani et al., 2017). En- sembling (Dietterich, 2000; Hansen and Salamon, 1990) of neural networks is a simple yet very ef- fective technique to improve the accuracy of NMT. The decoder makes use of K NMT networks which are either trained independently (Sutskever et al., 2014; Chung et al., 2016; Neubig, 2016; Wu et al., 2016) or share some amount of training iter- ations (Sennrich et al., 2016b,a; Cromieres et al., 2016; Durrani et al., 2017). The ensemble decoder computes predictions from each of the individual models which are then combined using the arith- metic average (Sutskever et al., 2014) or the geo- metric average (Cromieres et al., 2016). Ensembling consistently outperforms single NMT by a large margin. However, the decod- ing speed is significantly worse since the decoder needs to apply K NMT models rather than only one. Therefore, a recent line of research transfers the idea of knowledge distillation (Bucilu et al., 2006; Hinton et al., 2014) to NMT and trains a smaller network (the student) by minimizing the cross-entropy to the output of the ensemble system (the teacher) (Kim and Rush, 2016; Freitag et al., 2017). This paper presents an alternative to knowl- edge distillation as we aim to speed up decoding to be comparable to single NMT while retaining the boost in translation accuracy from the ensem- ble. In a first …

Content truncated to 500 words