Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2820–2825 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU Jacob Devlin jdevlin@microsoft.com Microsoft Research Abstract Attentional sequence-to-sequence models have become the new standard for ma- chine translation, but one challenge of such models is a significant increase in training and decoding cost compared to phrase-based systems. Here, we focus on efficient decoding, with a goal of achiev- ing accuracy close the state-of-the-art in neural machine translation (NMT), while achieving CPU decoding speed/through- put close to that of a phrasal decoder. We approach this problem from two an- gles: First, we describe several techniques for speeding up an NMT beam search de- coder, which obtain a 4.4x speedup over a very efficient baseline decoder with- out changing the decoder output. Sec- ond, we propose a simple but powerful network architecture which uses an RNN (GRU/LSTM) layer at bottom, followed by a series of stacked fully-connected lay- ers applied at every timestep. This ar- chitecture achieves similar accuracy to a deep recurrent model, at a small frac- tion of the training and decoding cost. By combining these techniques, our best system achieves a very competitive ac- curacy of 38.3 BLEU on WMT English- French NewsTest2014, while decoding at 100 words/sec on single-threaded CPU. We believe this is the best published ac- curacy/speed trade-off of an NMT system. 1 Introduction Attentional sequence-to-sequence models have become the new standard for machine transla- tion over the last two years, and with the un- precedented improvements in translation accuracy comes a new set of technical challenges. One of the biggest challenges is the high training and de- coding costs of these neural machine translation (NMT) system, which is often at least an order of magnitude higher than a phrase-based system trained on the same data. For instance, phrasal MT systems were able achieve single-threaded decod- ing speeds of 100-500 words/sec on decade-old CPUs (Quirk and Moore, 2007), while Jean et al. (2015) reported single-threaded decoding speeds of 8-10 words/sec on a shallow NMT system. Wu et al. (2016) was able to reach CPU decoding speeds of 100 words/sec for a deep model, but used 44 CPU cores to do so. There has been re- cent work in speeding up decoding by reducing the search space (Kim and Rush, 2016), but little in computational improvements. In this work, we consider a production sce- nario which requires low-latency, high-throughput NMT decoding. We focus on CPU-based de- coders, since GPU/FPGA/ASIC-based decoders require specialized hardware deployment and lo- gistical constraints such as batch processing. Ef- ficient CPU decoders can also be used for on- device mobile translation. We focus on single- threaded decoding and single-sentence processing, since multiple threads can be used to reduce la- tency but not total throughput. We approach this problem from two angles: In Section 4, we describe a number of techniques for improving the speed …

Content truncated to 500 words