Target Conditioning for One-to-Many Generation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2853–2862 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2853 Target Conditioning for One-to-Many Generation Marie-Anne Lachaux Facebook AI Research malachaux@fb.com Armand Joulin Facebook AI Research ajoulin@fb.com Guillaume Lample Facebook AI Research glample@fb.com Abstract Neural Machine Translation (NMT) models of- ten lack diversity in their generated transla- tions, even when paired with search algorithm, like beam search. A challenge is that the di- versity in translations are caused by the vari- ability in the target language, and cannot be in- ferred from the source sentence alone. In this paper, we propose to explicitly model this one- to-many mapping by conditioning the decoder of a NMT model on a latent variable that rep- resents the domain of target sentences. The domain is a discrete variable generated by a target encoder that is jointly trained with the NMT model. The predicted domain of target sentences are given as input to the decoder dur- ing training. At inference, we can generate di- verse translations by decoding with different domains. Unlike our strongest baseline (Shen et al., 2019), our method can scale to any num- ber of domains without affecting the perfor- mance or the training time. We assess the qual- ity and diversity of translations generated by our model with several metrics, on three differ- ent datasets. 1 Introduction Neural Machine Translation (NMT) models are trained to translate a sentence from a source lan- guage into a target language. There are many trans- lations of the same sentence that are both gram- matically correct and faithful to the source, but these translations may differ greatly in their vocab- ulary, style or grammar. Inferring the best transla- tion among them requires to explore a vast output space to cover this variability. This is typically handle as a post-processing step using a search algorithm, like beam search. This procedure is known to produce translations that lack in diversity, often differing only by a punctuation or a word (Ku- mar and Byrne, 2004; Li et al., 2016). While the search algorithm can certainly be improved, part of the problem resides also in the training of the NMT models; they are trained on 1-to-1 translation datasets without any objective to encourage diverse translations. There are many ways to model the diversity of translations from data that contain only one transla- tion, such as mixture of experts (Shen et al., 2019) or variational autoencoders (Zhang et al., 2016). A particularity of machine translation is that it is a one-to-many mapping problem. This means that the variability should be encoded by the target sen- tence and the question is how to combine a NMT system with a target sentence encoder with no pos- terior collapse. In this work, we propose to combine the encoder of the NMT with a discrete target encoder. Similar to other discrete autoencoders (Kaiser et al., 2018; van den Oord et al., 2017), each target sentence is assigned to a discrete variable, or …

Content truncated to 500 words