The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 445–449 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach John Duselis, Michael Hutt, Jeremy Gwinnup Air Force Research Laboratory {john.duselis,michael.hutt.ctr, jeremy.gwinnup.1}@us.af.mil James W. Davis Joshua Sandvick Ohio State University {davis.1719,sandvick.6}@osu.edu Abstract This paper introduces the AFRL-OSU Multimodal Machine Translation Task 1 system for submission to the Conference on Machine Translation 2017 (WMT17). This is an atypical MT system in that the image is the catalyst for the MT results, and not the textual content. 1 Introduction Contemporary scientific meetings have examined the potential benefits of fusing image information with machine translation. For instance, the leading international conference in this area, the Confer- ence on Machine Translation (WMT), is approach- ing its second year of competition on Multimodal Machine Translation (MMT). First year results in WMT16’s Multimodal Task 1 were varied in ap- proaches, informative in their findings, and indi- cated potential opprtunities for multimodal system improvement. (Specia et al., 2016). In the WMT16 submissions, the seemingly pre- dominant focal point across the systems was the fact that textual information was the driver for the translation. The image features tended towards be- ing ancilliary inputs or outputs (Libovický et al., 2016; Guasch and Costa-Jussà, 2016; Caglayan et al., 2016) or decision-type functions (Shah et al., 2016) and not the main antagonist for translation (Specia et al., 2016; Elliott et al., 2015). This is sensible as it is an MT competition. However, ap- proaching it from another direction, namely, hav- ing the image as the driver for the translation presents a different point of view worth investigat- ing. This work is sponsored by the Air Force Research Labo- ratory under AFRL/711 Human Performance Wing Chief Sci- entist Funding. The following sections will outline the seem- ingly novel approach to MMT and give particulars of this unconstrained system. 2 AFRL-OSU System This section will outline the architecture of the sys- tem. This is a first approximation into the pro- cess but is expected to undergo further develop- ment based on insights from this competition. 2.1 General Overview Referencing Fig. 1, a generic example taken from (Specia et al., 2016) shows a method where the source caption and image are the drivers for the multimodal translation. In some of WMT16’s sub- missions, the decomposition of the image is incor- porated as an additional feature into the MMT sys- tem, while others used the features as a function to help pick the best translation. AFRL-OSU’s system is pictorially represented in Figure 2. Currently, there is much work in image captioning systems (Socher et al., 2014; Ghahramani et al., 2014; Mao et al., 2014; Kiros et al., 2014; Vinyals et al., 2015), and WMT17 has even set out a task in its competition for it. Our em- phasis is not to try to produce a multilingual image captioning system, rather to use one to accomplish MT as the maturity of …

Content truncated to 500 words