A Dataset and Reranking Method for Multimodal MT of User-Generated Image Captions

A Dataset and Reranking Method for Multimodal MT of User-Generated Image Captions Shigehiko Schamoni schamoni@cl.uni-heidelberg.de Julian Hitschler hitschler@cl.uni-heidelberg.de Department of Computational Linguistics, Heidelberg University, 69120 Heidelberg, Germany Stefan Riezler riezler@cl.uni-heidelberg.de Department of Computational Linguistics and IWR, Heidelberg University, 69120 Heidelberg, Germany Abstract We present a dataset and method for improving the translation of noisy image captions that were created by users of Wikimedia Commons. The dataset is multilingual but non-parallel, and is several orders of magnitude larger than existing parallel data for multimodal machine translation. Our retrieval-based method pivots on similar images and uses the associated cap- tions in the target language to rerank translation outputs. This method only requires small amounts of parallel captions to find the optimal ensemble of retrieval features based on textual and visual similarity. Furthermore, our method is compatible with any machine translation system, and allows to quickly integrate new data without the need of re-training the translation system. Tests on three different datasets showed that size and diversity of the data is crucial for the performance of our method. On the introduced dataset we observe consistent improvements of up to 5 BLEU points and 3 points in Character F-score over strong neural MT baselines for three different language pairs. 1 Introduction Image caption translation is the task of translating a caption associated with an image into an- other language. What differentiates this task from purely text-based machine translation is the incorporation of image information into the translation process. Images associated with text usually add a new modality of information. Such information helps to ground the meaning of the corresponding text and is thus especially useful in a translation task. Interest in this task has surged since the first instantiation of the shared task on multimodal machine translation where a dataset of 30,000 German translations of crowdsourced English captions was presented (Spe- cia et al., 2016). However, this dataset has limitations: The captions were created by human annotators that were guided to produce “conceptual” descriptions that identify the objects de- picted in the image (Hodosh et al., 2013). This leads to relatively short captions amounting to a comparatively easy translation task with little room for improvement by incorporating visual information. This is confirmed by recent results showing that improvements over a text-only MT baseline are inconsistent and hard to achieve (Lala et al., 2017; Elliott et al., 2017). While caption translation in previous work has been conducted solely on clean, manually labeled captions based on MS COCO (Lin et al., 2014), Flickr30k (Rashtchian et al., 2010), or its multilingual variant Multi30k (Elliott et al., 2016), the goal of our work is to lift multimodal caption translation to a more realistic setup. For this purpose, we extracted a dataset of 4M “cap- Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 140 tions in the wild” as they appear in the user-generated Wikimedia Commons database. This new dataset is very different to previous image-caption data, as it contains …

Content truncated to 500 words