Learning Language-Specific Layers for Multilingual Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14767–14783 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Language-Specific Layers for Multilingual Machine Translation Telmo Pessoa Pires Robin M. Schmidt Yi-Hsiu Liao Stephan Peitz Apple {telmo, robin_schmidt, yihsiu_liao, speitz}@apple.com Abstract toMultilingual Machine Translation promises improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding losing gender and formality information when translating through English). On the downside, adding more languages reduces model capacity per language, which is usually countered by increasing the overall model size, making training harder and inference slower. In this work, we introduce Language-Specific Transformer Layers (LSLs), which allow us to increase model capacity, while keeping the amount of computation and the number of parameters used in the forward pass constant. The key idea is to have some layers of the encoder be source or target language-specific, while keeping the remaining layers shared. We study the best way to place these layers using a neural architecture search inspired approach, and achieve an improvement of 1.3 CHRF (1.5 SPBLEU) points over not using LSLs on a separate decoder architecture, and 1.9 CHRF (2.2 SPBLEU) on a shared decoder one. 1 Introduction Multilingual Neural Machine Translation (MNMT) has received much attention from the Machine Translation community in recent years (Johnson et al., 2017; Aharoni et al., 2019; Freitag and Firat, 2020; Zhang et al., 2020; Fan et al., 2021; Yang et al., 2021; Tran et al., 2021). This interest is based on the many advantages it provides: Scalability Instead of having one model per lan- guage pair, a single model suffices, significantly reducing maintenance efforts as well as the com- bined model size across all languages. Inference Speed and Less Error Cascading Due to the availability of data, most production sys- tems are English-centric, meaning translation be- tween two non-English languages naïvely involves translating twice (i.e. pivoting), once to English, and once from English. This approach increases latency and contributes to error cascades, since the translation to or from English usually implies in- formation loss, e.g. missing gender or formality distinctions that do not exist similarly in English. Low-Resource Improvements Having a single model capable of handling multiple languages, means it can generalize across language boundaries and utilize characteristics of closely related trans- lation directions to improve the translation quality for low-resource language-pairs (i.e. knowledge transfer). Although achieving good zero-shot trans- lation quality remains a challenging task, MNMT has been shown to help (Johnson et al., 2017). Despite the above advantages, training high quality multilingual models is a challenging task: as more languages are added, the more they compete for the model’s parameters (Sachan and Neubig, 2018). A common solution is to increase the model size, but blindly doing so comes with its own troubles, as training becomes harder, inference slower, and the storage requirements increase, which makes them challenging to deploy to portable devices. …

Content truncated to 500 words