Towards Multimodal Simultaneous Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 594–603 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 594 Towards Multimodal Simultaneous Neural Machine Translation Aizhan Imankulova∗ Masahiro Kaneko∗ Tosho Hirasawa∗ Mamoru Komachi Tokyo Metropolitan University 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan {imankulova-aizhan, kaneko-masahiro, hirasawa-tosho}@ed.tmu.ac.jp komachi@tmu.ac.jp Abstract Simultaneous translation involves translating a sentence before the speaker’s utterance is completed in order to realize real-time under- standing in multiple languages. This task is significantly more challenging than the gen- eral full sentence translation because of the shortage of input information during decoding. To alleviate this shortage, we propose mul- timodal simultaneous neural machine transla- tion (MSNMT), which leverages visual infor- mation as an additional modality. Our experi- ments with the Multi30k dataset showed that MSNMT significantly outperforms its text- only counterpart in more timely translation sit- uations with low latency. Furthermore, we verified the importance of visual information during decoding by performing an adversarial evaluation of MSNMT, where we studied how models behaved with incongruent input modal- ity and analyzed the effect of different word order between source and target languages. 1 Introduction Simultaneous translation is a natural language pro- cessing (NLP) task in which translation begins be- fore receiving the whole source sentence. It is widely used in international summits and confer- ences where real-time comprehension is one of the essential aspects. Simultaneous translation is already a difficult task for human interpreters be- cause the message must be understood and trans- lated while the input sentence is still incomplete, especially for language pairs with different word orders (e.g. SVO-SOV) (Seeber, 2015). Conse- quently, simultaneous translation is more challeng- ing for machines. Previous works attempt to solve this task by predicting the sentence-final verb (Gris- som II et al., 2014), or predicting unseen syntactic constituents (Oda et al., 2015). Given the difficulty ∗These authors contributed equally to this paper of predicting future inputs based on existing lim- ited inputs, Ma et al. (2019) proposed a simple simultaneous neural machine translation (SNMT) approach wait-k which generates the target sen- tence concurrently with the source sentence, but always k tokens behind, satisfying low latency re- quirements. However, previous approaches solve the given task by solely using the text modality, which may be insufficient to produce a reliable translation. Si- multaneous interpreters often consider various ad- ditional information sources such as visual clues or acoustic data while translating (Seeber, 2015). Therefore, we hypothesize that using supplemen- tary information, such as visual clues, can also be beneficial for simultaneous machine translation. To this end, we propose Multimodal Simul- taneous Neural Machine Translation (MSNMT) that supplements the incomplete textual modal- ity with visual information, in the form of an im- age. It will predict still missing information to improve translation quality during the decoding process. Our approach can be applied in various situations where visual information is related to the content of speech such as presentations with slides (e.g. TED Talks1) and news video broad- casts2. Our experiments show that the proposed MSNMT method …

Content truncated to 500 words