Monotonic Infinite Lookback Attention for Simultaneous Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1313–1323 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1313 Monotonic Infinite Lookback Attention for Simultaneous Machine Translation Naveen Arivazhagan∗ Colin Cherry∗ Wolfgang Macherey Chung-Cheng Chiu Semih Yavuz Google navari,colincherry,wmach,chungchengc@google.com syavuz,rpang,mweili,craffel@google.com Ruoming Pang Wei Li Colin Raffel Abstract Simultaneous machine translation begins to translate each source sentence before the source speaker is finished speaking, with ap- plications to live and streaming scenarios. Si- multaneous systems must carefully schedule their reading of the source sentence to bal- ance quality against latency. We present the first simultaneous translation system to learn an adaptive schedule jointly with a neural ma- chine translation (NMT) model that attends over all source tokens read thus far. We do so by introducing Monotonic Infinite Lookback (MILk) attention, which maintains both a hard, monotonic attention head to schedule the read- ing of the source sentence, and a soft attention head that extends from the monotonic head back to the beginning of the source. We show that MILk’s adaptive schedule allows it to ar- rive at latency-quality trade-offs that are fa- vorable to those of a recently proposed wait-k strategy for many latency values. 1 Introduction Simultaneous machine translation (MT) addresses the problem of how to begin translating a source sentence before the source speaker has finished speaking. This capability is crucial for live or streaming translation scenarios, such as speech-to- speech translation, where waiting for one speaker to complete their sentence before beginning the translation would introduce an intolerable delay. In these scenarios, the MT engine must balance latency against quality: if it acts before the nec- essary source content arrives, translation quality degrades; but waiting for too much source con- tent can introduce unnecessary delays. We refer to the strategy an MT engine uses to balance read- ing source tokens against writing target tokens as its schedule. ∗Equal contributions. Recent work in simultaneous machine transla- tion tends to fall into one of two bins: • The schedule is learned and/or adaptive to the current context, but assumes a fixed MT sys- tem trained on complete source sentences, as typified by wait-if-* (Cho and Esipova, 2016) and reinforcement learning approaches (Gris- som II et al., 2014; Gu et al., 2017). • The schedule is simple and fixed and can thus be easily integrated into MT training, as typi- fied by wait-k approaches (Dalvi et al., 2018; Ma et al., 2018). Neither scenario is optimal. A fixed schedule may introduce too much delay for some sentences, and not enough for others. Meanwhile, a fixed MT sys- tem that was trained to expect complete sentences may impose a low ceiling on any adaptive sched- ule that uses it. Therefore, we propose to train an adaptive schedule jointly with the underlying neu- ral machine translation (NMT) system. Monotonic attention mechanisms (Raffel et al., 2017; Chiu and Raffel, 2018) are designed for in- tegrated training in streaming scenarios and pro- vide our starting point. …

Content truncated to 500 words