CUNI Non-Autoregressive System for the WMT 22 Efficient Translation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 668 - 670 December 7-8, 2022 ©2022 Association for Computational Linguistics CUNI Non-Autoregressive System for the WMT 22 Efficient Translation Shared Task Jindˇrich Helcl Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostranské námˇestí 25, 11800 Prague, Czech Republic helcl@ufal.mff.cuni.cz Abstract We present a non-autoregressive system sub- mission to the WMT 22 Efficient Translation Shared Task. Our system was used by Helcl et al. (2022) in an attempt to provide fair com- parison between non-autoregressive and autore- gressive models. This submission is an effort to establish solid baselines along with sound evaluation methodology, particularly in terms of measuring the decoding speed. The model itself is a 12-layer Transformer model trained with connectionist temporal classification on knowledge-distilled dataset by a strong autore- gressive teacher model. 1 Introduction In the past few years, non-autoregressive (NAR) models for neural machine translation (NMT) at- tracted interest from the research community (Gu et al., 2018; Lee et al., 2018). Given the conditional independence between the output states, the decod- ing process can be parallelized across time steps. In theory, this leads to higher decoding speeds. Since efficient decoding is claimed to be the main motivation of non-autoregressive models, the Efficient Translation Shared Task seems to be the appropriate venue to provide fair comparison be- tween these models and their autoregressive coun- terparts. However, all submissions to this task were autoregressive so far (Birch et al., 2018; Hayashi et al., 2019; Heafield et al., 2020, 2021). Recently, Helcl et al. (2022) pointed out com- mon flaws in the evaluation methodology of NAR models. We found that optimized autoregressive models still achieve superior performance over NAR models. The only scenario where NAR mod- els showed some potential is GPU decoding with batch size of 1 (latency). Nevertheless, optimized autoregressive models were still both faster and better in terms of translation quality. The main pur- pose of this submission is to provide a reasonable baseline to future non-autoregressive submissions. 2 Model In our experiments, we use the non-autoregressive model proposed by Libovický and Helcl (2018) based on Connectionist Temporal Classification (CTC; Graves et al., 2006). We submit models that have been trained as a part of Helcl (2022). Architecture. The architecture is a 6-layer Trans- former encoder, followed by a state-splitting layer and another stack of 6 Transformer layers. The state-splitting layer takes the encoder states, project them into k-times wider states using an affine trans- formation, and then split the states into k-times longer sequence while retaining the original model dimension. In the submitted model, we set k = 3. The latter 6 layers cross-attend to the states imme- diately after state-splitting. We use Transformer model dimension of 1,024, 16 attention heads and a dimension of 4,096 in the feed-forward sublayer. The defining property of non-autoregressive models is that the decoding process treats output states as conditionally independent. In this archi- tecture, we set the output sequence length to k×Tx where Tx …

Content truncated to 500 words