Simultaneous Machine Translation with Visual Context

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2350–2361, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2350 Simultaneous Machine Translation with Visual Context Ozan Caglayan1, Julia Ive1, Veneta Haralampieva1, Pranava Madhyastha1 Lo¨ıc Barrault2 and Lucia Specia1,2,3 Imperial College London1, University of Sheffield2, ADAPT - Dublin City University3 o.caglayan@ic.ac.uk, j.ive@ic.ac.uk, vlh19@ic.ac.uk, pranava@ic.ac.uk l.barrault@sheffield.ac.uk, l.specia@ic.ac.uk Abstract Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. The translation thus has to start with an incomplete source text, which is read progressively, creating the need for anticipation. In this paper, we seek to understand whether the addition of visual information can compensate for the missing source context. To this end, we analyse the impact of different multimodal approaches and visual features on state-of-the-art SiMT frame- works. Our results show that visual context is helpful and that visually-grounded models based on explicit object region information are much better than commonly used global fea- tures, reaching up to 3 BLEU points improve- ment under low latency scenarios. Our quali- tative analysis illustrates cases where only the multimodal systems are able to translate cor- rectly from English into gender-marked lan- guages, as well as deal with differences in word order, such as adjective-noun placement between English and French. 1 Introduction Simultaneous machine translation (SiMT) aims to reproduce human interpretation, where an inter- preter translates spoken utterances as they are pro- duced. The interpreter has to dynamically find the balance between how much context is needed to generate the translation reliably, and how long the listener has to wait for the translation. In contrast to consecutive machine translation where source sentences are available in their entirety before trans- lation, the challenge in SiMT is thus the design of a strategy to find a good trade-off between the qual- ity of the translation and the latency incurred in producing it. Previous work has considered rule- based strategies that rely on waiting until some white chat blanc cat a un Figure 1: An illustration of a 1-word latency system that makes use of visual grounding to resolve the gen- der of the article ‘un’ and to predict the noun ‘chat’ af- ter reading its qualifier ‘white’. −� and −� denote READ and WRITE, respectively. constraint is satisfied, which includes approaches based on syntactic constraints (Bub et al., 1997; Ryu et al., 2006), segment/chunk/alignment infor- mation (Bangalore et al., 2012) heuristic-based con- ditions during decoding (Cho and Esipova, 2016) or deterministic policies with pre-determined la- tency constraints (Ma et al., 2019). An alternative line of research focuses on learning the decision policy: Gu et al. (2017) and Alinejad et al. (2018) frame SiMT as learning to generate READ/WRITE actions and employ reinforcement learning (RL) to formulate the problem as a policy agent interacting with its environment (i.e. a pre-trained MT model). Recent work has also explored supervised learning of the policy, by using oracle action sequences pre- dicted by a pre-trained …

Content truncated to 500 words