Generating Diverse Translation from Model Distribution with Dropout

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1088–1097, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1088 Generating Diverse Translation from Model Distribution with Dropout Xuanfu Wu1,2, Yang Feng1,2∗, Chenze Shao1,2 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China {wuxuanfu20s, fengyang, shaochenze18z}@ict.ac.cn Abstract Despite the improvement of translation qual- ity, neural machine translation (NMT) often suffers from the lack of diversity in its gener- ation. In this paper, we propose to generate diverse translations by deriving a large num- ber of possible models with Bayesian mod- elling and sampling models from them for in- ference. The possible models are obtained by applying concrete dropout to the NMT model and each of them has specific confidence for its prediction, which corresponds to a posterior model distribution under specific training data in the principle of Bayesian modeling. With variational inference, the posterior model dis- tribution can be approximated with a varia- tional distribution, from which the final mod- els for inference are sampled. We conducted experiments on Chinese-English and English- German translation tasks and the results shows that our method makes a better trade-off be- tween diversity and accuracy. 1 Introduction In the past several years, neural machine transla- tion (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Gehring et al., 2017; Vaswani et al., 2017; Zhang et al., 2019) based on the end-to-end model has achieved impressive progress in improving the accuracy of translation. Despite its remarkable success, NMT still faces problems in diversity. In natural language, due to lexical, syntactic and synonymous factors, there are usually multiple proper translations for a sentence. However, existing NMT models mostly implement one-to-one mapping between natural languages, that is, one source language sentence corresponds to one target language sentence. Although beam search, a widely used decoding algorithm, can gen- erate a group of translations, its search space is too narrow to extract diverse translations. ∗Corresponding author: Yang Feng There are some researches working at enhanc- ing translation diversity in recent years. Li et al. (2016) and Vijayakumar et al. (2016) proposed to add regularization terms to the beam search algo- rithm so that it can possess greater diversity. He et al. (2018) and Shen et al. (2019) introduced la- tent variables into the NMT model, thus the model can generate diverse outputs using different latent variables. Moreover, Sun et al. (2019) proposed to combine the structural characteristics of Trans- former and use the different weights between each head in the multi-head attention mechanism to ob- tain diverse results. In spite of improvement in balancing accuracy and diversity, these methods do not represent the diversity in the NMT model directly. In this paper, we take a different approach to gen- erate diverse translation by explicitly maintaining different models based on the principle of Bayesian Neural Networks (BNNs). These models are de- rived by applying concrete dropout (Gal et al., 2017) to …

Content truncated to 500 words