The Sockeye Neural Machine Translation Toolkit at AMTA 2018

The SOCKEYE Neural Machine Translation Toolkit at AMTA 2018 Felix Hieber, Tobias Domhan, Michael Denkowski {fhieber,domhant,mdenkows} @amazon.com David Vilar, Artem Sokolov, Ann Clifton, Matt Post {dvilar,artemsok,acclift,mattpost} @amazon.com Abstract We describe SOCKEYE,1 an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). SOCKEYE is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNET, the toolkit offers scalable training and inference for the three most prominent encoder- decoder architectures: attentional recurrent neural networks, self-attentional transformers, and fully convolutional networks. SOCKEYE also supports a wide range of optimizers, normalization and regularization techniques, and inference improvements from current NMT literature. Users can easily run standard training recipes, explore different model settings, and incorporate new ideas. The SOCKEYE toolkit is free software released under the Apache 2.0 license. 1 Introduction For all its success, Neural Machine Translation (NMT) presents a range of new challenges. While popular encoder-decoder models are attractively simple, recent literature and the results of shared evaluation tasks show that a significant amount of engineering is required to achieve “production-ready” performance in both translation quality and computational efficiency. In a trend that carries over from Statistical Machine Translation (SMT), the strongest NMT systems benefit from subtle architecture modifications, hyper-parameter tuning, and empirically effective heuristics. To address these challenges, we introduce SOCKEYE, a neural sequence-to-sequence toolkit written in Python and built on Apache MXNET2 [Chen et al., 2015]. To the best of our knowledge, SOCKEYE is the only toolkit that includes implementations of all three major neural translation architectures: attentional recurrent neural networks [Schwenk, 2012, Kalchbrenner and Blunsom, 2013, Sutskever et al., 2014, Bahdanau et al., 2014, Luong et al., 2015], self- attentional transformers [Vaswani et al., 2017], and fully convolutional networks [Gehring et al., 2017]. These implementations are supported by a wide and continually updated range of features reflecting the best ideas from recent literature. Users can easily train models based on the latest research, compare different architectures, and extend them by adding their own code. SOCKEYE is under active development that follows best practice for both research and production software, including clear coding and documentation guidelines, comprehensive automatic tests, and peer review for code contributions. 1https://github.com/awslabs/sockeye (version 1.12) 2https://mxnet.incubator.apache.org/ Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 200 2 Encoder-Decoder Models for NMT The main idea in neural sequence-to-sequence modeling is to encode a variable-length input sequence of tokens into a sequence of vector representations, and to then decode those representa- tions into a sequence of output tokens. We give a brief description for the three encoder-decoder architectures as implemented in SOCKEYE, but refer the reader to the references for more details or to the arXiv version of this paper [Hieber et al., 2017]. 2.1 Stacked Recurrent Neural Network (RNN) with Attention [Bahdanau et al., 2014, Luong et al., 2015] The encoder consists of a bi-directional RNN followed by a stack of uni-directional RNNs. An RNN at layer l produces …

Content truncated to 500 words