THUMT: An Open-Source Toolkit for Neural Machine Translation

THUMT: An Open-Source Toolkit for Neural Machine Translation Zhixing Tan†, Jiacheng Zhang†, Xuancheng Huang†, Gang Chen†, Shuo Wang†, Maosong Sun†‡, Huanbo Luan†, and Yang Liu†‡ ∗ †Institute for Artificial Intelligence Department of Computer Science and Technology, Tsinghua University Beijing National Research Center for Information Science and Technology ‡Beijing Academy of Artificial Intelligence Abstract THUMT is an open-source toolkit for neural machine translation (NMT) developed by the Natural Language Processing Group at Tsinghua University. The toolkit is easy to use, mod- ify and extend while provides the latest advances in NMT research and production. THUMT implements several standard NMT models and supports distributed training across multiple ma- chines, fast inference, and model visualization. Experiments on English-German and Chinese- English datasets show that THUMT can obtain results that are comparable to state-of-the-art NMT systems. 1 Introduction Machine translation (MT), which investigates the use of computers to translate human lan- guages automatically, is an important task in natural language processing and artificial intel- ligence communities. With the availability of bilingual machine-readable texts, data-driven approaches to machine translation have gained wide popularity since the 1990s (Hutchins and Lovtskii, 2000). Recent several years have witnessed the rapid development of end-to-end neu- ral machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017). Capable of learning representations from data, NMT has quickly replaced conventional statistical machine translation (SMT) (Brown et al., 1993; Koehn et al., 2003; Chiang, 2005) to become the new de facto method in practical MT systems (Wu et al., 2016). This paper introduces THUMT, an open-source NMT toolkit targeting both academia and industry. THUMT originally developed with Theano (Theano Development Team, 2016) and begins its launch in June 2017. With the emerging of new deep learning frameworks, THUMT added TensorFlow (Abadi et al., 2016) implementation in October 2017 and PyTorch (Paszke et al., 2019) implementation in August 2019. The current status of the three implementations are as follows: • THUMT-Theano (Zhang et al., 2017): the original project developed with Theano, which is no longer updated because MLA put an end to Theano. It implemented the stan- dard attention-based model (RNNsearch) (Bahdanau et al., 2014), minimum risk train- ing (MRT) (Shen et al., 2015) for optimizing model parameters with respect to evaluation metrics, semi-supervised training (SST) (Cheng et al., 2016) for exploiting monolingual ∗Corresponding author. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 116 Features Theano TensorFlow PyTorch Models RNNsearch Seq2Seq, RNNsearch, Transformer Transformer Criterions MLE, MRT, SST MLE MLE Optimizers SGD, Adadelta, Adam Adam SGD, Adadelta, Adam LRP Yes Yes No Gradient Aggregation No Yes Yes Distributed Training No Yes Yes Mixed-Precision No Yes Yes TensorBoard No Yes Yes Table 1: Available features in different implementations. corpora to learn bi-directional translation models, and layer-wise relevance propagation (LRP) (Ding et al., 2017) for visualizing and analyzing RNNsearch. • THUMT-TensorFlow: an implementation focuses on performance. It implemented the sequence-to-sequence model (Seq2Seq) (Sutskever et …

Content truncated to 500 words