Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1235–1244 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1235 Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation Ye Liu1, Yao Wan2, Jian-Guo Zhang1, Wenting Zhao1, Philip S. Yu1 1Department of Computer Science, University of Illinois at Chicago, Chicago, IL, USA 2School of Computer Sci. & Tech., Huazhong University of Science and Technology, Wuhan, China {yliu279, jzhan51, wzhao41, psyu}@uic.edu, wanyao@hust.edu.cn Abstract The non-autoregressive models have boosted the efficiency of neural machine translation through parallelized decoding at the cost of effectiveness, when comparing with the au- toregressive counterparts. In this paper, we claim that the syntactic and semantic struc- tures among natural language are critical for non-autoregressive machine translation and can further improve the performance. How- ever, these structures are rarely considered in existing non-autoregressive models. Inspired by this intuition, we propose to incorporate the explicit syntactic and semantic structures of languages into a non-autoregressive Trans- former, for the task of neural machine transla- tion. Moreover, we also consider the interme- diate latent alignment within target sentences to better learn the long-term token dependen- cies. Experimental results on two real-world datasets (i.e., WMT14 En-De and WMT16 En- Ro) show that our model achieves a signifi- cantly faster speed, as well as keeps the transla- tion quality when compared with several state- of-the-art non-autoregressive models. 1 Introduction Recently, non-autoregressive models (Gu et al., 2018), which aim to enable the parallel genera- tion of output tokens without sacrificing translation quality, have attracted much attention. Although the non-autoregressive models have considerably sped up the inference process for real-time neural machine translation (NMT) (Gu et al., 2018), their performance is considerably worse than that of autoregressive counterparts. Most previous works attribute the poor performance to the inevitable con- ditional independence issue when predicting target tokens, and many variants have been proposed to solve it. For example, several techniques in non- autoregressive models are investigated to mitigate the trade-off between speedup and performance, including iterative refinement (Lee et al., 2018), insertion-based models (Chan et al., 2019; Stern et al., 2019), latent variable based models (Kaiser et al., 2018; Shu et al., 2020), CTC models (Li- bovick´y and Helcl, 2018; Saharia et al., 2020), al- ternative loss function based models (Wei et al., 2019; Wang et al., 2019; Shao et al., 2020), and masked language models (Ghazvininejad et al., 2019, 2020). Although these works have tried to narrow the performance gap between autoregressive and non- autoregressive models, and have achieved some improvements on machine translation, the non- autoregressive models still suffer from syntactic and semantic limitations. That is, the translations of non-autoregressive models tend to contain inco- herent phrases (e.g., repetitive words), and some informative tokens on the source side are absent. It is because in non-autoregressive models, each token in the target sentence is generated indepen- dently. Consequently, it will cause the multimodal- ity issue, i.e., the non-autoregressive models can- not model the multimodal …

Content truncated to 500 words