Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 341–356 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation Aditi Jain∗ IIT Delhi mt6190739@iitd.ac.in Nishant Kambhatla∗ School of Computing Science Simon Fraser University nkambhat@sfu.ca Anoop Sarkar School of Computing Science Simon Fraser University anoop@sfu.ca Abstract The decoder in simultaneous neural machine translation receives limited information from the source while having to balance the oppos- ing requirements of latency versus translation quality. In this paper, we use an auxiliary target- side language model to augment the training of the decoder model. Under this notion of target adaptive training, generating rare or difficult tokens is rewarded which improves the transla- tion quality while reducing latency. The predic- tions made by a language model in the decoder are combined with the traditional cross entropy loss which frees up the focus on the source side context. Our experimental results over multiple language pairs show that compared to previous state of the art methods in simultaneous trans- lation, we can use an augmented target side context to improve BLEU scores significantly. We show improvements over the state of the art in the low latency range with lower average lagging values (faster output). 1 1 Introduction Simultaneous Machine Translation (SiMT; Gris- som II et al. (2014); Cho and Esipova (2016)) is a special case of neural machine translation (NMT; Vaswani et al. (2017)) that aims to produce real time-translations in the target language from a streaming input in the source language. The cor- nerstone of this task, as well as a key challenge, is the trade-off between the translation quality and the latency in producing the translations. This bal- ance is ensured by a fixed (Ma et al., 2019; El- bayad et al., 2020) or adaptive (Arivazhagan et al., 2019; Ma et al., 2020; Zhang and Feng, 2022b) read/write policy that determines whether to wait for the next source token (a READ action) or to generate a translation (a WRITE action). Adaptive policies dynamically predict the action based on 1github.com/sfu-natlang/target_rescale_siMT ∗Equal contribution. Listing order is random. Work performed while AJ was visiting SFU Natlang Lab. It 's quite warm in winter , but summer is extremely hot . <EOS> 0.0 0.1 0.2 It was cozy in winter but extremely hot in summer . LM MMA Reads 3210 4 1 0.7 1.2 SiMT policy Source Target conditional max. likelihood language model target context token importance 1 1 1 rescaling ours prior work Figure 1: Prior work on simultaneous MT weighs ev- ery target token equally. Top: Normalized negative log-likelihood (nll) scores of each generated in-context target token as scored by the baseline SiMT along with the number of reads preceding a target token, and a target language model (LM). As the translations are im- perfect, the LM shows disagreement by following an opposite nll trend compared to the translation model. Bottom: Our method rescales the importance of each target token using the …

Content truncated to 500 words