Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1613–1624 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1613 Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders Xiang Kong‡∗, Adithya Renduchintala†, James Cross†, Yuqing Tang†, Jiatao Gu†, Xian Li† ‡Language Technologies Institute, Carnegie Mellon University †Facebook AI xiangk@cs.cmu.edu {adirendu,jcross,yuqtang,jgu,xianl}@fb.com Abstract Recent work in multilingual translation ad- vances translation quality surpassing bilin- gual baselines using deep transformer mod- els with increased capacity. However, the ex- tra latency and memory costs introduced by this approach may make it unacceptable for efficiency-constrained applications. It has re- cently been shown for bilingual translation that using a deep encoder and shallow decoder (DESD) can reduce inference latency while maintaining translation quality, so we study similar speed-accuracy trade-offs for multilin- gual translation. We find that for many-to- one translation we can indeed increase de- coder speed without sacrificing quality us- ing this approach, but for one-to-many trans- lation, shallow decoders cause a clear qual- ity drop. To ameliorate this drop, we pro- pose a deep encoder with multiple shallow de- coders (DEMSD) where each shallow decoder is responsible for a disjoint subset of target languages. Specifically, the DEMSD model with 2-layer decoders is able to obtain a 1.8x speedup on average compared to a standard transformer model with no drop in translation quality. 1 Introduction Encoder-decoder based neural machine translation (NMT) systems have achieved great success on bilingual translation tasks (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017). Recently, mul- tilingual neural machine translation (MNMT) has also attracted much attention because of its ease of deployment, knowledge transfer among lan- guages and the potential to enable zero-shot trans- lation (Dong et al., 2015; Firat et al., 2016; Ha et al., 2016; Johnson et al., 2017; Arivazhagan et al., ∗Work done at Facebook AI. 2019; Zhang et al., 2020). While MNMT can sup- port translations in several directions, not all of them have better performance when compared to their corresponding bilingual models. Suspecting that poor performance in some directions is due to the limited model capacity, many prior works adopt deeper encoder and decoder (Zhang et al., 2019; Wang et al., 2019; Zhang et al., 2020). However, increasing the number of layers, especially in the decoder, deteriorates the latency of translation and memory costs. Recently, Kasai et al. (2020) show that given a fixed capacity budget, as measured by the number of layers, models with a deep encoder and a shallow decoder (DESD) are faster at infer- ence time when compared to standard models with an equal number of encoder and decoder layers while maintaining translation quality. Inspired by findings from Kasai et al. (2020), in this work, we explore the speed-accuracy trade-off in multilingual machine translation systems. Given the same model capacity budget, we experiment various layer allocation strategies. We analyze mul- tilingual models in the one-to-many (O2M) setting and many-to-one (M2O) …

Content truncated to 500 words