Efficient Machine Translation with Model Pruning and Quantization

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 775–780 November 10–11, 2021. ©2021 Association for Computational Linguistics 775 Efficient Machine Translation with Model Pruning and Quantization Maximiliana Behnke† Nikolay Bogoychev† Alham Fikri Aji† Kenneth Heafield† Graeme Nail† Qianqian Zhu† Svetlana Tchistiakova† Jelmer van der Linde† Pinzhen Chen† Sidharth Kashyap‡ Roman Grundkiewicz†§ †University of Edinburgh ‡Intel Corporation §Microsoft {maximiliana.behnke, n.bogoych, a.fikri, kenneth.heafield, graeme.nail, qianqian.zhu, stchisti, jelmer.vanderlinde, pinzhen.chen, rgrundki}@ed.ac.uk, sidharth.n.kashyap@intel.com Abstract We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strate- gies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical for- mats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions op- timize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, em- phasizing component- and block-level pruning that actually improves speed unlike coefficient- wise pruning. 1 Introduction This paper describes the University of Edinburgh’s submission to Sixth Conference on Machine Trans- lation (WMT2021) Efficiency Task1, which mea- sures performance on latency and throughput on both CPU and GPU, in addition to translation qual- ity. Our submission focused on the trade-off be- tween these metrics and quality. Our submission builds upon the work of last year’s submission (Bogoychev et al., 2020). We trained our models in a teacher-student setting (Kim and Rush, 2016), using Edinburgh’s En-De system submitted to the WMT2021 news transla- tion task as the teacher model. For the students, we used a Simpler Simple Recurrent Unit (SSRU) (Kim et al., 2019) decoder, used a target vocabu- lary shortlist, and experimented with pruning the student models by removing component- and block- level parameters to improve speed. We further ex- perimented with quantizing into smaller numerical 1http://statmt.org/wmt21/ efficiency-task.html formats, including fixed point 8-bit quantization on the CPU, and both 8-bit and log based 4-bit quan- tization on the GPU, as well as post-quantization fine-tuning of 4-bit quantized models. For running our experiments, we improved upon the Marian (Junczys-Dowmunt et al., 2018) ma- chine translation framework by incorporating speed ups for 8-bit matrix multiplication operations, opti- mizations for pruning neural network parameters on Intel CPUs, and exploring tensorcores on the GPU. 1.1 Efficiency Shared Task The WMT21 efficiency shared task consists of two sub-tasks: throughput and latency. Systems should translate English to German under the constrained conditions of the WMT21 news task. For each task, systems are provided 1 million lines of raw English input with at most 150 space-separated words. The throughput task receives this input directly. The latency task, introduced this year, is fed input one sentence at a time, waiting for the translation output before providing the next sentence. Throughput is measured on multi-core CPU or GPU system, and latency is measured on single- core …

Content truncated to 500 words