Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training

Proceedings of NAACL-HLT 2019, pages 3857–3867 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3857 Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training Tasnim Mohiuddin⋆ and Shafiq Joty⋆† ⋆Nanyang Technological University, Singapore †Salesforce Research Asia, Singapore {mohi0004,srjoty}@ntu.edu.sg Abstract Adversarial training has shown impressive success in learning bilingual dictionary with- out any parallel data by mapping monolingual embeddings to a shared space. However, re- cent work has shown superior performance for non-adversarial methods in more challenging language pairs. In this work, we revisit ad- versarial autoencoder for unsupervised word translation and propose two novel extensions to it that yield more stable training and im- proved results. Our method includes regu- larization terms to enforce cycle consistency and input reconstruction, and puts the target encoders as an adversary against the corre- sponding discriminator. Extensive experimen- tations with European, non-European and low- resource languages show that our method is more robust and achieves better performance than recently proposed adversarial and non- adversarial approaches. 1 Introduction Learning cross-lingual word embeddings has been shown to be an effective way to transfer knowl- edge from one language to another for many key linguistic tasks including machine translation, named entity recognition, part-of-speech tagging, and parsing (Ruder et al., 2017). While earlier ef- forts solved the associated word alignment prob- lem using large parallel corpora (Luong et al., 2015), broader applicability demands methods to relax this requirement since acquiring a large cor- pus of parallel data is not feasible in most sce- narios. Recent methods instead use embeddings learned from monolingual data, and learn a linear mapping from one language to another with the underlying assumption that two embedding spaces exhibit similar geometric structures (i.e., approx- imately isomorphic). This allows the model to learn effective cross-lingual representations with- out expensive supervision (Artetxe et al., 2017). Given monolingual word embeddings of two languages, Mikolov et al. (2013a) show that a lin- ear mapping can be learned from a seed dictio- nary of 5000 word pairs by minimizing the sum of squared Euclidean distances between the mapped vectors and the target vectors. Subsequent works (Xing et al., 2015; Artetxe et al., 2016, 2017; Smith et al., 2017) propose to improve the model by normalizing the embeddings, imposing an or- thogonality constraint on the mapper, and modi- fying the objective function. While these methods assume some supervision in the form of a seed dic- tionary, recently fully unsupervised methods have shown competitive results. Zhang et al. (2017a,b) first reported encouraging results with adversar- ial training. Conneau et al. (2018) improved this approach with post-mapping refinements, show- ing impressive results for several language pairs. Their learned mapping was then successfully used to train a fully unsupervised neural machine trans- lation system (Lample et al., 2018a,b). Although successful, adversarial training has been criticized for not being stable and failing to converge, inspiring researchers to propose non- adversarial methods more recently (Xu et al., 2018a; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018; Artetxe …

Content truncated to 500 words