Sparse and Decorrelated Representations for Stable Zero-shot NMT

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2260–2266 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2260 Sparse and Decorrelated Representations for Stable Zero-shot NMT Bokyung Son1,2, Sungwon Lyu1 1Kakao Enterprise / Seoul, Republic of Korea 2Department of Linguistics, Seoul National University / Seoul, Republic of Korea {meta.mon, james.ryu}@kakaoenterprise.com Abstract Using a single encoder and decoder for all directions and training with English-centric data is a popular scheme for multilingual NMT. However, zero-shot translation under this scheme is vulnerable to changes in train- ing conditions, as the model degenerates by de- coding non-English texts into English regard- less of the target specifier token. We present that enforcing both sparsity and decorrelation on encoder intermediate representations with the SLNI regularizer (Aljundi et al., 2019) effi- ciently mitigates this problem, without perfor- mance loss in supervised directions. Notably, effects of SLNI turns out to be irrelevant to pro- moting language-invariance in encoder repre- sentations. 1 Introduction As massive language pairs are supported in recent works in neural machine translation (NMT) (Aha- roni et al., 2019; Arivazhagan et al., 2019b), obtain- ing training data becomes more of an issue. Due to limited availability of parallel corpora, datasets for multilingual NMT are in many cases English- centric— English is either on the source side or the target side—, or at least missing several pairs among the supported language set. This leads to a conspicuous need for a model to support zero-shot translation, which is to translate between language pairs for which no parallel training data exists. A popular scheme for multilingual NMT is to have one encoder and one decoder shared across all trained directions, and prepend a reserved token to the source text to indicate the target language. This model is capable of zero-shot translation; setting the target token which was unpaired with the source at training time still works (Wu et al., 2016; Ha et al., 2016; Johnson et al., 2017). However, while being parameter-efficient, an exposure bias arises when trained with English-centric data; as non- English languages are always trained to be trans- lated into English, they are wrongly decoded into English for zero-shot directions (Ha et al., 2016, 2017). In fact, zero-shot NMT under this scheme is extremely sensitive to hyperparmeters including batch size, dropout, and weight initialization (Gu et al., 2019). Fixing the hyperparameters favor- able to zero-shot directions would not be desirable, however, if such conditions hurt performance on supervised directions. We utilize the Sparse coding through Local Neu- ral Inhibition (SLNI) (Aljundi et al., 2019) regular- izer to make the representations more robust to hy- perparameters. SLNI was originally suggested as a continual learning technique by enforcing represen- tation sparsity and decorrelation. Here, we deviate from its previous use and focus on its single-stage effects during joint multitask training of multiple language pairs. We present that enforcing represen- tation sparsity and decorrelation together stabilizes zero-shot performance across various training con- ditions, without hurting performance on supervised directions. 2 Related Work Gu et …

Content truncated to 500 words