Locality-Sensitive Hashing for Long Context Neural Machine Translation

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 32 - 42 May 26-27, 2022 c⃝2022 Association for Computational Linguistics Locality-Sensitive Hashing for Long Context Neural Machine Translation Frithjof Petrick Jan Rosendahl Christian Herold Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany surname@i6.informatik.rwth-aachen.de Abstract After its introduction, the Transformer archi- tecture (Vaswani et al., 2017) quickly became the gold standard for the task of neural ma- chine translation. A major advantage of the Transformer compared to previous architec- tures is the faster training speed achieved by complete parallelization across timesteps due to the use of attention over recurrent layers. How- ever, this also leads to one of the biggest prob- lems of the Transformer, namely the quadratic time and memory complexity with respect to the input length. In this work we adapt the locality-sensitive hashing approach of Kitaev et al. (2020) to self-attention in the Transformer, we extended it to cross-attention and apply this memory efficient framework to sentence- and document-level machine translation. Our ex- periments show that the LSH attention scheme for sentence-level comes at the cost of slightly reduced translation quality. For document-level NMT we are able to include much bigger con- text sizes than what is possible with the base- line Transformer. However, more context does neither improve translation quality nor improve scores on targeted test suites. 1 Introduction After its introduction in 2017, the Transformer ar- chitecture (Vaswani et al., 2017) quickly became the gold standard for the task of neural machine translation (NMT) (Ott et al., 2018). Furthermore, variants of the Transformer have since been used very successfully for a variety of other tasks such as language modeling (LM) (Irie et al., 2019), nat- ural language understanding (NLU) (Devlin et al., 2019; Liu et al., 2019), speech translation (ST) (Vila et al., 2018), automatic speech recognition (ASR) (Zeyer et al., 2019; Mohamed et al., 2019) and image processing (Parmar et al., 2018). A major advantage of the Transformer com- pared to previous architectures is the faster training speed achieved by complete parallelization across timesteps. However, this also leads to one of the biggest problems of the Transformer, namely the quadratic time and memory complexity of atten- tion layers with respect to the sequence length. For sentence-level NMT this is not a big issue as most of the time the length of sequences is relatively short and can be handled efficently, even if sub- word segmentation is applied (Sennrich et al., 2016; Kudo, 2018). However, this drastically changes when moving towards character-level (Gupta et al., 2019) or document-level (Tiedemann and Scherrer, 2017) NMT. Especially for the latter, speed and memory issues are one of the biggest roadblocks towards ‘true’ document level systems (Junczys- Dowmunt, 2019). This leads to the situation where most works make do with including just a few sentences as a form of ‘local’ context information (Tiedemann and Scherrer, 2017; Jean et al., 2017; Bawden et al., 2018) or heavily …

Content truncated to 500 words