Sequence to Sequence Mixture Model for Diverse Machine Translation

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 583–592 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 583 Sequence to Sequence Mixture Model for Diverse Machine Translation Xuanli He Gholamreza Haffari Mohammad Norouzi Monash University, Australia Google Brain {xuanli.he1, gholamreza.haffari}@monash.edu mnorouzi@google.com Abstract Sequence to sequence (SEQ2SEQ) models of- ten lack diversity in their generated transla- tions. This can be attributed to the limitation of SEQ2SEQ models in capturing lexical and syntactic variations in a parallel corpus result- ing from different styles, genres, topics, or am- biguity of the translation process. In this pa- per, we develop a novel sequence to sequence mixture (S2SMIX) model that improves both translation diversity and quality by adopting a committee of specialized translation models rather than a single translation model. Each mixture component selects its own training dataset via optimization of the marginal log- likelihood, which leads to a soft clustering of the parallel corpus. Experiments on four lan- guage pairs demonstrate the superiority of our mixture model compared to a SEQ2SEQ base- line with standard or diversity-boosted beam search. Our mixture model uses negligible ad- ditional parameters and incurs no extra com- putation cost during decoding. 1 Introduction Neural sequence to sequence (SEQ2SEQ) models have been remarkably effective machine transla- tion (MT) (Sutskever et al., 2014; Bahdanau et al., 2015). They have revolutionized MT by provid- ing a unified end-to-end framework, as opposed to the traditional approaches requiring several sub- models and long pipelines. The neural approach is superior or on-par with statistical MT in terms of translation quality on various MT tasks and do- mains e.g. (Wu et al., 2016; Hassan et al., 2018). A well recognized issue with SEQ2SEQ mod- els is the lack of diversity in the generated trans- lations. This issue is mostly attributed to the de- coding algorithm (Li et al., 2016), and recently to the model (Zhang et al., 2016; Schulz et al., 2018a). The former direction has attempted to design diversity encouraging decoding algorithm, particularly beam search, as it generates transla- tions sharing the majority of their tokens except a few trailing ones. The latter direction has investi- gated modeling enhancements, particularly the in- troduction of continuous latent variables, in order to capture lexical and syntactic variations in train- ing corpora, resulted from the inherent ambiguity of the human translation process.1 However, im- proving the translation diversity and quality with SEQ2SEQ models is still an open problem, as the results of the aforementioned previous work are not fully satisfactory. In this paper, we develop a novel sequence to sequence mixture (S2SMIX) model that improves both translation quality and diversity by adopt- ing a committee of specialized translation models rather than a single translation model. Each mix- ture component selects its own training dataset via optimization of the marginal log-likelihood, which leads to a soft clustering of the parallel corpus. As such, our mixture model introduces a conditioning global discrete latent variable for each sentence, which leads to …

Content truncated to 500 words