Multimodal Lexical Translation

Multimodal Lexical Translation Chiraag Lala and Lucia Specia Department of Computer Science, University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK {clala1, l.specia}@sheffield.ac.uk Abstract Inspired by the tasks of Multimodal Machine Translation and Visual Sense Disambiguation we introduce a task called Multimodal Lexical Translation (MLT). The aim of this new task is to correctly translate an ambiguous word given its context - an image and a sentence in the source language. To facilitate the task, we introduce the MLT dataset, where each data point is a 4-tuple consisting of an ambiguous source word, its visual context (an image), its textual context (a source sentence), and its translation that conforms with the visual and textual contexts. The dataset has been created from the Multi30K corpus using word-alignment followed by human inspection for translations from English to German and English to French. We also introduce a simple heuristic to quantify the extent of the ambiguity of a word from the distribution of its translations and use it to select subsets of the MLT Dataset which are difficult to translate. These form a valuable multimodal and multilingual language resource with several potential uses including evaluation of lexical disambiguation within (Multimodal) Machine Translation systems. Keywords: Multimodal Machine Translation, Visual Sense Disambiguation, Multimodal Multilingual Language Resources 1. Introduction Multimodal Machine Translation is the task of translating text using information in other modalities (such as images) as auxiliary cues. It has been recently framed as a shared task as part of the last two editions of the Conference on Machine Translation (WMT16, WMT17) (Specia et al., 2016; Elliott et al., 2017). Within the Conference on Ma- chine Translation, the task is defined as: Given an image and its description in the source language, the objective is to translate the description into a target language, where this process can be supported by information from the image, as depicted in Figure 1. Figure 1: Multimodal Machine Translation Shared Task One of the main motivations to introduce multimodality in Machine Translation is the intuition that information from other modalities could help find the correct sense of am- biguous words in the source sentence, which could poten- tially lead to more accurate translations. For example, the English sentence “A man is holding a seal” could have at least two different translations in German depending on the sense of the word seal - (1) “Ein Mann h¨alt ein Siegel”, and (2) “Ein Mann h¨alt einen Seehund”. The images (Figure 2) could help a Multimodal Machine Translation system dis- ambiguate the correct sense of the word seal and translate accordingly. Disambiguation of word senses, popularly known as Word Sense Disambiguation or Lexical Disambiguation, is a (a) Ein Mann h¨alt ein Siegel (b) Ein Mann h¨alt einen Seehund Figure 2: Two different translations of “A man is holding a seal” depending on the visual context widely studied natural language processing task. Given an ambiguous word and its context, the objective is to assign the correct sense of the word based on …

Content truncated to 500 words