Token Drop mechanism for Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4298–4303 Barcelona, Spain (Online), December 8-13, 2020 4298 Token Drop mechanism for Neural Machine Translation Huaao Zhang Soochow University / Suzhou China hazhang@stu.suda.edu.cn Xiangyu Duan∗ Soochow University / Suzhou China xiangyuduan@suda.edu.cn Shigui Qiu Soochow University / Suzhou China sgqiu@stu.suda.edu.cn Min Zhang Soochow University / Suzhou China minzhang@suda.edu.cn Abstract Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Sim- ilar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our ap- proach and our model achieves significant improvements over a strong Transformer baseline1. 1 Introduction Neural machine translation (NMT) achieved enormous success in advancing the quality of translation (Bahdanau et al., 2015; Vaswani et al., 2017; Gehring et al., 2017). In spite of the impressive perfor- mance, NMT models are still vulnerable to perturbations in the input sentences (Belinkov and Bisk, 2018; Cheng et al., 2019) , i.e. a tiny perturbation will affect hidden representation and lead to low quality of translation (Zhao et al., 2018). Moreover, NMT commonly consists of millions of parameters, which making it prone to overfitting especially in low resource scene. A natural way to improve generalization is synthesizing natural noise (Karpukhin et al., 2019) or adopting arbitrary noise (Cheng et al., 2018; Ebrahimi et al., 2018). Another way is exploring regular- ization techniques to avoid overfitting (Miceli et al., 2017), making model robust to unseen or unfamiliar inputs. However, as discrete data, the text is hard to retain the semantic information after corruption. In this paper, we propose Token Drop to prevent overfitting and improve generalization. Different from standard dropout (Srivastava et al., 2014) that drops neurons in network randomly, we drop tokens of the input sentences. In order to retain semantic information, we replace tokens with a special symbol < unk > . This allows model learn hidden representation from rest token’s context, and predict target translation condition on latent variable. On the one hand, our method allows model meeting exponen- tially different sentences can be explained as data augmentation; On the other hand, our method corrupts input sentences with natural noise can be seen as regularization term for NMT. We investigate two self-supervised objectives: Replaced Token Detection and Dropped Token Predic- tion. Considering our Token Drop method regularize parameters by weakening model inputs, making NMT suitable for applying self-supervised objective. During training: (1) use a discriminator to detect whether input tokens are dropped or not; (2) leverage hidden state to predict original tokens of dropped tokens inspired by Cloze task (Devlin et al., 2019). Both of them guide model to generate semantically similar …

Content truncated to 500 words