CluBERT: A Cluster-Based Approach for Learning Sense Distributions in Multiple Languages

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376–385 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 376 Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation Junliang Guo Linli Xu∗ Enhong Chen Anhui Province Key Laboratory of Big Data Analysis and Application, School of Computer Science and Technology, University of Science and Technology of China guojunll@mail.ustc.edu.cn {linlixu,cheneh}@ustc.edu.cn Abstract The masked language model has received re- markable attention due to its effectiveness on various natural language processing tasks. However, few works have adopted this tech- nique in the sequence-to-sequence models. In this work, we introduce a jointly masked sequence-to-sequence model and explore its application on non-autoregressive neural ma- chine translation (NAT). Specifically, we first empirically study the functionalities of the en- coder and the decoder in NAT models, and find that the encoder takes a more important role than the decoder regarding the translation quality. Therefore, we propose to train the encoder more rigorously by masking the en- coder input while training. As for the decoder, we propose to train it based on the consecu- tive masking of the decoder input with an n- gram loss function to alleviate the problem of translating duplicate words. The two types of masks are applied to the model jointly at the training stage. We conduct experiments on five benchmark machine translation tasks, and our model can achieve 27.69/32.24 BLEU scores on WMT14 English-German/German-English tasks with 5+ times speed up compared with an autoregressive model. 1 Introduction The encoder-decoder based sequence-to-sequence framework (Sutskever et al., 2014; Bahdanau et al., 2014) has achieved great success on the task of Neural Machine Translation (NMT) (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Hassan et al., 2018; Sheng et al., 2020). In this framework, the encoder takes the source sentence as input and extracts its hidden representation, based on which the decoder generates the target sentence word by word and from left to right, i.e., ∗Corresponding author. in an autoregressive manner, which is a natural bot- tleneck for the inference speed due to the sequential conditional dependence. As the performance of NMT models have been substantially promoted, the translation effi- ciency is becoming a new research hotspot. Non- autoregressive neural machine translation (NAT) models are proposed to reduce the translation la- tency while inference, by removing the conditional dependence between target tokens and predicting all tokens in parallel (Gu et al., 2017). As the con- text dependency cannot be utilized while decoding, the inference speedup of NAT models comes at the cost of the degradation in performance. As studied by previous works (Guo et al., 2019; Wang et al., 2019), the inferior accuracy of NAT models mainly occurs from two aspects: 1) the source-side infor- mation is not adequately encoded which results in incomplete translation; 2) the decoder cannot handle the task well which leads to repeated trans- lations and poor performance on long sentences. To tackle these problems and promote the per- formance of NAT …

Content truncated to 500 words