Revisiting Modularized Multilingual NMT to Meet Industrial Demands

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5905–5918, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5905 Revisiting Modularized Multilingual NMT to Meet Industrial Demands Sungwon Lyu1, Bokyung Son1,2, Kichang Yang1,3, Jaekyoung Bae1 1Kakao Enterprise / Seoul, Republic of Korea 2Department of Linguistics, Seoul National University 3School of Software, Soongsil University {james.ryu, meta.mon, kevin.y, storm.b}@kakaoenterprise.com Abstract The complete sharing of parameters for mul- tilingual translation (1-1) has been the main- stream approach in current research. However, degraded performance due to the capacity bot- tleneck and low maintainability hinders its ex- tensive adoption in industries. In this study, we revisit the multilingual neural machine trans- lation model that only share modules among the same languages (M2) as a practical al- ternative to 1-1 to satisfy industrial require- ments. Through comprehensive experiments, we identify the benefits of multi-way training and demonstrate that the M2 can enjoy these benefits without suffering from the capacity bottleneck. Furthermore, the interlingual space of the M2 allows convenient modification of the model. By leveraging trained modules, we find that incrementally added modules exhibit better performance than singly trained models. The zero-shot performance of the added mod- ules is even comparable to supervised models. Our findings suggest that the M2 can be a com- petent candidate for multilingual translation in industries. 1 Introduction With the current increase in the demand for neural machine translation (NMT), serving an increasing number of languages poses a practical problem for the industry. A naive approach for multilingual NMT is to have multiple single-directional models, which is unsustainable owing to the quadratic in- crease of models as more languages are introduced. A more practical approach is to limit the number of models by sharing the components among the models (Dong et al., 2015; Firat et al., 2016a; Ha et al.; Johnson et al., 2017). In addition to reducing the number of parameters, sharing the components is also regarded as an effective method to enhance the performance. A fully shared model (henceforth 1-1), which only uses one encoder and one decoder to translate all directions (Ha et al.; Johnson et al., 2017), has been the most popular method because of its compactness. However, introduction of a significant number of tasks into a 1-1 model is known to cause capacity bottleneck. Aharoni et al. (2019) suggested that, given a fixed model capacity, a 1-1 model is bound to the tradeoff between the number of languages and translation accuracy. Zhang et al. (2020) explic- itly identified the capacity bottleneck problem of the 1-1 model by showing a clear decrease in per- formance when translation directions are doubled. Moreover, data unbalance complicates the problem. Arivazhagan et al. (2019b) presented the transfer and interference dilemma among low and high re- source languages in an unbalanced environment. The capacity bottleneck observed in the 1-1 model is particularly undesirable for the industry. Unlimited scaling of the model size (Zhang et al., 2020) is impossible in practice, where inference cost and latency are crucial. With …

Content truncated to 500 words