Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8226–8237 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8226 Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting Po-Yao Huang1, Junjie Hu1, Xiaojun Chang2, Alexander Hauptmann1 1Language Technologies Institute, Carnegie Mellon University 2Faculty of Information Technology, Monash University {poyaoh, junjieh, alex}@cs.cmu.edu, cxj273@gmail.com Abstract Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sen- tences in the latent space. As people speak different languages biologically share simi- lar visual systems, the potential of achiev- ing better alignment through visual content is promising yet under-explored in unsuper- vised multimodal MT (MMT). In this pa- per, we investigate how to utilize visual con- tent for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually- pivoted captioning as additional weak supervi- sion. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of- the-art methods and generalizes well when im- ages are not available at the testing time. 1 Introduction Neural machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) has achieved near human-level performance (Wu et al., 2016). However, its effectiveness strongly relies on the availability of large-scale parallel corpora. Unfortunately, preparing the parallel data remains a challenge as there are more than 6,500 languages in the world, and recruiting translators with bilin- gual or multilingual knowledge to cover all those languages is impractical. As a result, developing methods alleviating the need of well-annotated large parallel corpora has recently attracted increasing attention in the com- munity. These methods fall into two broad cate- gories. The first type of methods use a third lan- guage as the pivot (Firat et al., 2016; Chen et al., 2017; Cheng et al., 2017; Johnson et al., 2017) to enable zero-resource translation. Although the progress is encouraging, pivoting with a third lan- guage still demands bilingual knowledge for col- lecting large-scale parallel source-pivot and pivot- target corpora. The second type of methods explore unsupervised approaches (Conneau et al., 2018a; Artetxe et al., 2018; Lample et al., 2018a) have recently achieved impressive translation quality. These methods rely only on monolingual data and back-translation (Sennrich et al., 2016a). However, as discussed in (Lample et al., 2018b), the align- ment of source-target sentences is uncertain and highly subject to proper initialization. Using visual content for unsupervised MT (Chen et al., 2018; Su et al., 2019) is a promising solu- tion for pivoting and alignment based on its avail- ability and feasibility. Abundant multimodal con- tent in various languages are available online (e.g. Instagram and YouTube). It is also easier to re- cruit monolingual annotators to describe an image than to find multilingual translators to translate sentences. Importantly, visual content is eligible to improve the alignments in the language latent spaces since …

Content truncated to 500 words