HW-TSC’s Participation in the WMT 2021 Efficiency Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 781–786 November 10–11, 2021. ©2021 Association for Computational Linguistics 781 HW-TSC’s Participation in the WMT 2021 Efficiency Shared Task Hengchao Shang1, Ting Hu2, Daimeng Wei1, Zongyao Li1, Jianfei Feng2, Zhengzhe Yu1, Jiaxin Guo1, Shaojun Li1, Lizhi Lei1, Shimin Tao1, Hao Yang1, Jun Yao2, Ying Qin1, 1Huawei Translation Service Center, Beijing, China 2Huawei Noah’s Ark Lab, Hong Kong, China {shanghengchao,huting35,weidaimeng,lizongyao, fengjianfei1,yuzhengzhe,guojiaxin1,lishaojun15, leilizhi,taoshimin,yanghao30,yaojun97,qinying}@huawei.com Abstract This paper presents the submission of Huawei Translation Services Center (HW-TSC) to WMT 2021 Efficiency Shared Task. We ex- plore the sentence-level teacher-student distil- lation technique and train several small-size models that find a balance between efficiency and quality. Our models feature deep en- coder, shallow decoder and light-weight RNN with SSRU layer. We use Huawei Noah’s Bolt1, an efficient and light-weight library for on-device inference. Leveraging INT8 quan- tization, self-defined General Matrix Multi- plication (GEMM) operator, shortlist, greedy search and caching, we submit four small- size and efficient translation models with high translation quality for the one CPU core la- tency track. 1 Introduction Transformer and its variants (Vaswani et al., 2017; Shaw et al., 2018; So et al., 2019; Dehghani et al., 2019) have become benchmark models in the do- main of machine translation. A lot of innovations and engineering optimizations (Tay et al., 2020) in this area are based on Transformer. In general, to train a high-quality translation model, a large amount of data is required. Expensive training and deployment costs pose great challenges to scenar- ios where hardware are limited or the deployment environment is complex. This task aims to explore a solution that balances efficient decoding and high- quality translation. We focus on the one CPU la- tency track, which can better demonstrate the capa- bility of our model and inference framework. We explore a balance between speed and quality, and ensure efficient memory usage and light-weight in- ference framework capability at the same time. We finally submit four models of different sizes. We use knowledge distillation (Hinton et al., 2015) to train small models. The teacher mod- 1https://github.com/huawei-noah/bolt els come from our WMT 2021 News Shared Task. For the sake of efficient decoding, our models have only 1-layer decoder. However, the number of en- coding layers vary. Such settings lead to a great increase of inference efficiency while ensuring the translation quality (Wang et al., 2019). All of our experiments are conducted based on fariseq (Ott et al., 2019), including the training of teacher and student models, as well as the genera- tion of distillation data. We use Huawei Noah’s Bolt as the inference library. Bolt is a universal deep learning library featuring light weight and high speed. For the CPU task, we realize INT8 quantization inference and efficient GEMM operator, which is faster than Intel oneDNN2. With other engineering optimization strategies, we achieve a significant improvement in terms of inference efficiency. Section 2 describes the teacher-student knowl- edge distillation process. Section 3 introduces how we optimize inference for this task. Section …

Content truncated to 500 words