DePA: Improving Non-autoregressive Translation with Dependency-Aware Decoder

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 478–490 July 13-14, 2023 c⃝2023 Association for Computational Linguistics DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder Jiaao Zhan1, Qian Chen, Boxing Chen, Wen Wang, Yu Bai1, Yang Gao1∗ 1School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China jiaao_zhan@163.com {lukechan1231,chenboxing,wwang.969803}@gmail.com {yubai,gyang}@bit.edu.cn Abstract Non-autoregressive machine translation (NAT) models have lower translation quality than au- toregressive translation (AT) models because NAT decoders do not depend on previous tar- get tokens in the decoder input. We propose a novel and general Dependency-Aware De- coder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input. First, we propose an autore- gressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training. Sec- ond, we transform the decoder input from the source language representation space to the tar- get language representation space through a novel attentive transformation process, which enables the decoder to better capture target de- pendencies. DePA can be applied to any fully NAT models. Extensive experiments show that DePA consistently improves highly competi- tive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.1 1 Introduction Autoregressive translation (AT) systems achieve state-of-the-art (SOTA) performance for neural ma- chine translation (NMT) and Transformer (Vaswani et al., 2017) encoder-decoder is the prevalent ar- chitecture. In AT systems, each generation step depends on previously generated tokens, resulting in high inference latency when output is long. Non- autoregressive translation (NAT) models (Gu et al., 2018) significantly accelerate inference by gener- ating all target tokens independently and simulta- neously. However, this independence assumption ∗Corresponding Author 1We released our code at: https://github.com/ zhanjiaao/NAT_DePA. leads to degradation in accuracy compared to AT models, as NAT models cannot properly learn tar- get dependencies. Dependency in prior works and our work takes its standard definition in NLP, i.e., syntactic relations between words in a sentence. The mainstream NAT models fall into two cate- gories: iterative NAT models and fully NAT models. Iterative NAT models (Gu et al., 2019; Ghazvinine- jad et al., 2019; Lee et al., 2018) improve transla- tion accuracy by iteratively refining translations at the expense of slower decoding speed. In contrast, fully NAT models (Gu et al., 2018; Bao et al., 2022) have great latency advantage over AT models by making parallel predictions with a single decod- ing round, but they suffer from lower translation accuracy. In this paper, we aim at improving the translation accuracy of fully NAT models while preserving their latency advantage. Previous research (Gu and Kong, 2021) argues that reducing dependencies is crucial for training a fully NAT model effectively, as it allows the model to more easily capture target dependencies. How- ever, dependency reduction limits the performance upper bound of fully NAT models, since models …

Content truncated to 500 words