Using Visual Feature Space as a Pivot Across Languages

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3673–3678 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3673 Using Visual Feature Space as a Pivot Across Languages Ziyan Yang1 Leticia Pinto-Alva1,2 Franck Dernoncourt3 Vicente Ordonez1 1University of Virginia, 2San Pablo Catholic University, 3Adobe Research {zy3cx,lp2rv,vicente}@virginia.edu, dernonco@adobe.com Abstract Our work aims to leverage visual feature space to pass information across languages. We show that models trained to generate textual captions in more than one language condi- tioned on an input image can leverage their jointly trained feature space during inference to pivot across languages. We particularly demonstrate improved quality on a caption generated from an input image, by leveraging a caption in a second language. More impor- tantly, we demonstrate that even without condi- tioning on any visual input, the model demon- strates to have learned implicitly to perform to some extent machine translation from one lan- guage to another in their shared visual feature space. We show results in German-English, and Japanese-English language pairs that pave the way for using the visual world to learn a common representation for language. 1 Introduction There has been great interest in learning visual rep- resentations from images paired with natural lan- guage annotations. While tasks such as image cap- tion generation e.g. (Young et al., 2014; Lin et al., 2014) have focused mostly on English text, there is a growing body of work extending to a larger set of languages (Calixto et al., 2012; Elliott et al., 2015, 2016). Images annotated in multiple languages of- fer the possibility of studying grounded models of languages along with their commonalities and in- trinsics in direct connection with the visual world. We focus in the multilingual image description generation setting where we train an image encoder with soft-attention (Xu et al., 2015) and multiple text decoders for each target language. Then, we demonstrate that information from one language can be transferred to another language using energy based inference (LeCun et al., 2006) in an iterative fashion by leveraging the backpropagation algo- rithm at test time. Effectively, we demonstrate that a group of men are fishing on a beach drei Männer in einem Ruderboot Visual Feature Space a brown dog runs after a black dog on a shore zwei Hunde spielen auf dem Strand girl hits a ball and the catcher looks on ein Schiedsrichter beobachtet zwei Baseballspieler Figure 1: Our work shows how visual features capture multi-lingual information in image conditioned mod- els (solid blue arrows) and how to pivot this informa- tion across languages during inference by incorporat- ing feedback connections (dotted red arrows) from lan- guage back to visual feature space. the common visual feature space used to generate text in the target languages also learns implicitly alignments between them and thus acts as its own form of “visual language”. Figure 1 shows some example images and textual descriptions in Ger- man and English, as well as a general outline of our approach. We demonstrate our findings by (1) showing …

Content truncated to 500 words