Cross-lingual Visual Verb Sense Disambiguation

Proceedings of NAACL-HLT 2019, pages 1998–2004 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1998 Cross-lingual Visual Verb Sense Disambiguation Spandana Gella∗ and Desmond Elliott† and Frank Keller∗ ∗School of Informatics, University of Edinburgh †Department of Computer Science, University of Copenhagen {spandana.gella,frank.keller}@ed.ac.uk, de@di.ku.dk Abstract Recent work has shown that visual context im- proves cross-lingual sense disambiguation for nouns. We extend this line of work to the more challenging task of cross-lingual verb sense disambiguation, introducing the Multi- Sense dataset of 9,504 images annotated with English, German, and Spanish verbs. Each im- age in MultiSense is annotated with an English verb and its translation in German or Span- ish. We show that cross-lingual verb sense dis- ambiguation models benefit from visual con- text, compared to unimodal baselines. We also show that the verb sense predicted by our best disambiguation model can improve the re- sults of a text-only machine translation system when used for a multimodal translation task. 1 Introduction Resolving lexical ambiguity remains one of the most challenging problems in natural language pro- cessing. It is often studied as a word sense dis- ambiguation (WSD) problem, which is the task of assigning the correct sense to a word in a given context (Kilgarrif, 1998). Word sense disambigua- tion is typically tackled using only textual context; however, in a multimodal setting, visual context is also available and can be used for disambigua- tion. Most prior work on visual word sense dis- ambiguation has targeted noun senses (Barnard and Johnson, 2005; Loeff et al., 2006; Saenko and Darrell, 2008), but the task has recently been ex- tended to verb senses (Gella et al., 2016, 2019). Resolving sense ambiguity is particularly crucial for translation tasks, as words can have more than one translation, and these translations often corre- spond to word senses (Carpuat and Wu, 2007; Nav- igli, 2009). As an example consider the verb ride, which can translate into German as fahren (ride a bike) or reiten (ride a horse). Recent work on mul- timodal machine translation has partly addressed Source: Three guys riding on an elephant. Target: Drei M¨anner reiten auf einem Elefanten. Output: Drei Jungs fahren auf einem Elefanten. Figure 1: An example of a verb sense translation er- ror (shown in bold red) by the English-German neural translation system of Sennrich et al. (2017). lexical ambiguity by using visual information, but it still remains unresolved especially for the part-of- speech categories such as verbs (Specia et al., 2016; Shah et al., 2016; Hitschler et al., 2016; Lala and Specia, 2018). Prior work on cross-lingual WSD has been limited in scale and has only employed textual context (Lefever and Hoste, 2013), even though the task should benefit from visual context, just like monolingual WSD. Visual information has been shown to be use- ful to map words across languages for bilingual lexicon induction. For this, images are used as a pivot between languages or visual information is combined with cross-lingual vector spaces to learn word translations …

Content truncated to 500 words