Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1461–1467, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1461 Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations Po-Yao Huang1, Xiaojun Chang2, Alexander Hauptmann1 1Language Technologies Institute, Carnegie Mellon University 2Faculty of Information Technology, Monash University poyaoh@cs.cmu.edu, cxj273@gmail.com, alex@cs.cmu.edu Abstract With the aim of promoting and understand- ing the multilingual version of image search, we leverage visual object detection and pro- pose a model with diverse multi-head atten- tion to learn grounded multilingual multi- modal representations. Specifically, our model attends to different types of textual semantics in two languages and visual objects for fine- grained alignments between sentences and im- ages. We introduce a new objective func- tion which explicitly encourages attention di- versity to learn an improved visual-semantic embedding space. We evaluate our model in the German-Image and English-Image match- ing tasks on the Multi30K dataset, and in the Semantic Textual Similarity task with the En- glish descriptions of visual content. Results show that our model yields a significant per- formance gain over other methods in all of the three tasks. 1 Introduction Joint visual-semantic embeddings (VSE) are cen- tral to the success of many vision-language tasks, including cross-modal search and retrieval (Kiros et al., 2014; Karpathy and Fei-Fei, 2015; Gu et al., 2018), visual question answering (Antol et al., 2015; Goyal et al., 2017), multimodal machine translation (Huang et al., 2016; Elliott and K´ad´ar, 2017), etc. Learning VSE requires extensive un- derstanding of the content in individual modalities and an in-depth alignment strategy to associate the complementary information from multiple views. With the availability of large-scale parallel English-Image corpora (Lin et al., 2014; Young et al., 2014), a rich line of research has advanced learning VSE under the monolingual setup. Most recent works (Kiros et al., 2014; Vendrov et al., 2015; Karpathy and Fei-Fei, 2015; Klein et al., 2015; Wang et al., 2016, 2018; Huang et al., 2019b) leverage triplet ranking losses to align En- glish sentences and images in the joint embedding space. In VSE++ (Faghri et al., 2018), Faghri et al. improve VSE by emphasizing hard negative samples. Recent advancement in VSE models ex- plores methods to enrich the English-Image cor- pora. Shi et al. (2018) propose to augment dataset with textual contrastive adversarial samples to combat adversarial attacks. Recently, Huang et al. (2019a) utilize textual semantics of regional ob- jects and adversarial domain adaptation for learn- ing VSE under low-resource constraints. An emerging trend generalizes learning VSE in the multilingual scenario. Rajendran et al. (2016) learn M-view representations when parallel data is available only between one pivot view and the rest of views. PIVOT (Gella et al., 2017) extends the work from Calixto et al. (2017) to use images as the pivot view for learning multilingual multi- modal representations. K´ad´ar et al. (2018) further confirms the benefits of multilingual training. Our work is motivated by Gella et …

Content truncated to 500 words