The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020

The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020 Tobias Domhan domhant@amazon.com Michael Denkowski mdenkows@amazon.com David Vilar dvilar@amazon.com Xing Niu xingniu@amazon.com Felix Hieber fhieber@amazon.com Amazon Kenneth Heafield∗ translate@kheafield.com Efficient Translation Limited Abstract We present Sockeye 2, a modernized and streamlined version of the Sockeye neural machine translation (NMT) toolkit. New features include a simplified code base through the use of MXNet’s Gluon API, a focus on state of the art model architectures, distributed mixed precision training, and efficient CPU decoding with 8-bit quantization. These improvements result in faster training and inference, higher automatic metric scores, and a shorter path from research to production. 1 Introduction Sockeye (Hieber et al., 2017) is a versatile toolkit for research in the fast-moving field of NMT. Since the initial release, it has been used in at least 25 scientific publications, including winning submissions to WMT evaluations (Schamper et al., 2018). Sockeye also powers Amazon Trans- late, showing industrial-strength performance in addition to the flexibility needed in academic environments. Moreover, we are excited to see that hardware manufacturers are contributing to optimizing MXNet (Chen et al., 2015) and Sockeye for speed. Intel has demonstrated large performance gains for Sockeye inference on Intel Skylake processors.1 NVIDIA is working on significant performance improvements for Sockeye’s Transformer (Vaswani et al., 2017) implementation through fused operators and an optimized beam search. This paper discusses Sockeye 2’s streamlined Gluon implementation (§2), support for state of the art architectures and efficient decoding (§3), and improved model training (§4). 2 Gluon Implementation Sockeye 2 is implemented using Gluon,2 MXNet’s latest and preferred API that combines the strengths of imperative and graph-based programming. Gluon provides a simple Python inter- face and uses eager execution by default, allowing developers to quickly prototype deep learning models and debug issues step by step. At runtime, Gluon can automatically “hybridize” models ∗Work was done in an external advisory capacity. 1https://www.intel.ai/amazing-inference-performance-with-intel-xeon- scalable-processors/#gs.wrgsji 2https://mxnet.apache.org/versions/1.6/api/python/docs/api/gluon/index.html Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 110 DE–EN EN–DE FI–EN EN–FI Layers BLEU Latency BLEU Latency BLEU Latency BLEU Latency 6:6 35.5 602 37.9 791 22.2 575 20.5 808 10:10 35.4 970 37.8 1238 22.3 863 20.8 1258 20:2 34.8 293 37.6 357 23.2 257 20.9 368 Table 1: SacreBLEU scores (Post, 2018) and single-sentence latency in milliseconds on new- stest2019 for models trained on WMT19 constrained data with varying numbers of encoder and decoder layers. Latency values are the 90th percentile of translation time when translating each sentence individually (no batching). We measure single sentence decoding latency on an EC2 c5.2xlarge instance with 4 CPU cores. We report the average over three independent training runs. by converting them into computation graphs for maximum performance. Adopting this pro- gramming model significantly simplifies Sockeye 2’s training and inference code, reducing the overall Python line count by 25%. Sockeye 2’s hybridized transformer also improves training speed by 14% compared to Sockeye. 3 Focus on State of the …

Content truncated to 500 words