LIUM-CVC Submissions for WMT17 Multimodal Translation Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 432–439 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics LIUM-CVC Submissions for WMT17 Multimodal Translation Task Ozan Caglayan†, Walid Aransa, Adrien Bardet, Mercedes Garc´ıa-Mart´ınez, Fethi Bougares, Lo¨ıc Barrault LIUM, University of Le Mans †ozancag@gmail.com FirstName.LastName@univ-lemans.fr Marc Masana, Luis Herranz and Joost van de Weijer CVC, Universitat Autonoma de Barcelona {joost,mmasana,lherranz}@cvc.uab.es Abstract This paper describes the monomodal and multimodal Neural Machine Translation systems developed by LIUM and CVC for WMT17 Shared Task on Multimodal Translation. We mainly explored two mul- timodal architectures where either global visual features or convolutional feature maps are integrated in order to benefit from visual context. Our final systems ranked first for both En→De and En→Fr language pairs according to the automatic evaluation metrics METEOR and BLEU. 1 Introduction With the recent advances in deep learning, purely neural approaches to machine translation, such as Neural Machine Translation (NMT), (Sutskever et al., 2014; Bahdanau et al., 2014) have received a lot of attention because of their competitive per- formance (Toral and S´anchez-Cartagena, 2017). Another reason for the popularity of NMT is its flexible nature allowing researchers to fuse auxil- iary information sources in order to design sophis- ticated networks like multi-task, multi-way and multi-lingual systems to name a few (Luong et al., 2015; Johnson et al., 2016; Firat et al., 2017). Multimodal Machine Translation (MMT) aims to achieve better translation performance by vi- sually grounding the textual representations. Re- cently, a new shared task on Multimodal Machine Translation and Crosslingual Image Captioning (CIC) was proposed along with WMT16 (Specia et al., 2016). In this paper, we present MMT sys- tems jointly designed by LIUM and CVC for the second edition of this task within WMT17. Last year we proposed a multimodal atten- tion mechanism where two different attention dis- tributions were estimated over textual and im- age representations using shared transformations (Caglayan et al., 2016a). More specifically, convo- lutional feature maps extracted from a ResNet-50 CNN (He et al., 2016) pre-trained on the ImageNet classification task (Russakovsky et al., 2015) were used to represent visual information. Although our submission ranked first among multimodal sys- tems for CIC task, it was not able to improve over purely textual NMT baselines in neither tasks (Specia et al., 2016). The winning submission for MMT (Caglayan et al., 2016a) was a phrase-based MT system rescored using a language model en- riched with FC7 global visual features extracted from a pre-trained VGG-19 CNN (Simonyan and Zisserman, 2014). State-of-the-art results were obtained after WMT16 by using a separate attention mecha- nism for different modalities in the context of CIC (Caglayan et al., 2016b) and MMT (Calixto et al., 2017a). Besides experimenting with multimodal attention, Calixto et al. (2017a) and Libovick´y and Helcl (2017) also proposed a gating extension in- spired from Xu et al. (2015) which is believed to allow the decoder to learn when to attend to a particular modality although Libovick´y and Helcl (2017) report no …

Content truncated to 500 words