TenTrans High-Performance Inference Toolkit for WMT2021 Efficiency Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 795–798 November 10–11, 2021. ©2021 Association for Computational Linguistics 795 TenTrans High-Performance Inference Toolkit for WMT2021 Efficiency Task Kaixin Wu, Bojie Hu, Qi Ju∗ TencentMT Oteam {danielkxwu, bojiehu, damonju}@tencent.com Abstract The paper describes the TenTrans’s submis- sions to the WMT 2021 Efficiency Shared Task. We explore training a variety of smaller compact transformer models using the teacher- student setup. Our model is trained by our self-developed open-source multilingual train- ing platform TenTrans-Py1. We also release an open-source high-performance inference toolkit2 for transformer models and the code is written in C++ completely. All additional optimizations are built on top of the inference engine including attention caching, kernel fu- sion, early-stop, and several other optimiza- tions. In our submissions, the fastest system can translate more than 22,000 tokens per sec- ond with a single Tesla P4 while maintaining 38.36 BLEU on En-De newstest2019. Our trained models and more details are available in TenTrans-Decoding competition examples3. 1 Introduction We participate in the GPU throughput track of the Workshop on Machine Translation (WMT) 2021 Efficiency Shared Task. The efficiency task aims at exploring the different techniques for training and optimizing GPU models for high throughput while preserving the highest possible accuracy. While we do not pay more attention to training techniques, we apply a variety of optimizations to improve the computation efficiency of our GPU models in the inference phase. In terms of the training phase, we trained a va- riety of smaller compact student models using the common teacher-student training approach (Hinton et al., 2015; Kim and Rush, 2016) on our open- source multilingual training platform TenTrans- ∗ Corresponding author. 1https://github.com/TenTrans/TenTrans 2https://github.com/TenTrans/ TenTrans-Decoding 3https://github.com/TenTrans/ TenTrans-Decoding/blob/master/examples/ WMT21-Efficiency.md Py. All of them are based on the deep trans- former which has proven more effective and has lower training costs than the wide transformer mod- els (Wang et al., 2019). For the inference phase, our strategy for the shared task includes attention caching, kernel fusion, early-stop, and several other optimizations. All of these optimizations are em- ployed in a high-optimized and C++-based infer- ence engine TenTrans-Decoding. The paper is structured as follows: Section 2 describes the data preparation and the training de- tails, then Section 3 presents the variety of ours optimizations to improve decoding efficiency. The detailed accuracy and efficiency results are shown in Section 4. Finally, we conclude our work in Section 5. 2 Teacher-student Training To train smaller compact student models, the teacher-student training approach (Hinton et al., 2015; Kim and Rush, 2016) is adopted. First, a large model (the teacher) is trained on all avail- able bilingual data, included synthetic data gener- ated by the back-translation (Sennrich et al., 2015a) method. Multiple model ensembles are also typi- cally used to build stronger teacher systems. Then, all our small optimized models (the student) are created using sequence-level knowledge distilla- tion (Kim and Rush, 2016) and trained on data generated from the teacher model. The sequence- level knowledge distillation is a common technique that …

Content truncated to 500 words