Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 100 - 108 December 7-8, 2022 ©2022 Association for Computational Linguistics Findings of the WMT 2022 Shared Task on Efficient Translation Kenneth Heafield Biao Zhang Graeme Nail Jelmer van der Linde Nikolay Bogoychev University of Edinburgh 10 Crichton Street Edinburgh, Scotland EH8 9AB {Kenneth.Heafield,b.zhang,graeme.nail,jelmer.vanderlinde,n.bogoych}@ed.ac.uk Abstract The machine translation efficiency task chal- lenges participants to make their systems faster and smaller with minimal impact on transla- tion quality. How much quality to sacrifice for efficiency depends upon the application, so participants were encouraged to make multi- ple submissions covering the space of trade- offs. In total, there were 76 submissions from 5 teams. The task covers GPU, single-core CPU, and multi-core CPU hardware tracks as well as batched throughput or single-sentence latency conditions. Submissions showed hundreds of millions of words can be translated for a dollar, average latency is 3.5–25 ms, and models fit in 7.5–900 MB. 1 Introduction The efficiency task complements the collocated news task by challenging participants to make their machine translation systems computationally effi- cient. This is the fifth edition of the task, expanding upon previous editions (Heafield et al., 2021, 2020; Hayashi et al., 2019; Birch et al., 2018). Participants built English→German machine translation systems following a constrained data condition. The data condition follows the con- strained 2021 Workshop on Machine Translation news translation task. This year, to reduce the barrier to entry, organisers provided an ensemble of teacher systems, as well as cleaned data and distilled output from the teacher ensemble. Partic- ipants were required to use the provided teacher systems, but were free to distil additional data from the constrained condition. The SentencePiece vo- cabulary used by the teachers was also made avail- able. For translation quality measurement, we use the news-focused WMT22 dataset, and the systems are ranked according to the COMET (Rei et al., 2020) automatic metric. We also evaluate systems on BLEU and chrF for additional reference. Throughput Latency CPU-ALL GPU CPU-1 GPU CUNI 1 1 1 1 ECNU 1 1 1 1 Edinburgh 15 11 15 11 HuaweiTSC 5 5 RoyalFlush 6 Table 1: Number of systems submitted by each partici- pant for the different hardware and batching conditions. CPU-ALL refers to the 36-core hardware setting. Submissions are made as Docker containers so we can consistently measure their performance in terms of quality, speed, memory usage, and disk space. We run the containers in three different hardware environments: one GPU, one CPU core, and multiple CPU cores. Systems were tested for throughput by providing 1 million sentences up- front to allow batching and parallelization. We also tested for latency with a program that drip-feeds one input sentence, waits for the translation, and then provides the next input sentence. There were four conditions in total: GPU throughput, GPU Latency, 1 CPU Core Latency, and 36 CPU cores throughput. We did not measure latency in a multi- core CPU setting because the test hardware has 36 cores and overhead for 36 threads …

Content truncated to 500 words