Too Brittle to Touch: Comparing the Stability of Quantization and Distillation towards Developing Low-Resource MT Models

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 870 - 885 December 7-8, 2022 ©2022 Association for Computational Linguistics Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models Harshita Diddee1 Sandipan Dandapat2 Monojit Choudhury1 Tanuja Ganu1 Kalika Bali1 1 Microsoft Research, India 2 Microsoft R&D, India {t-hdiddee,sadandap,monojitc,taganu,kalikab}@microsoft.com Abstract Leveraging shared learning through Massively Multilingual Models, state-of-the-art machine translation (MT) models are often able to adapt to the paucity of data for low-resource lan- guages. However, this performance comes at the cost of significantly bloated models which are not practically deployable. Knowledge Dis- tillation is one popular technique to develop competitive lightweight models: In this work, we first evaluate it’s use to compress MT mod- els focusing specifically on languages with ex- tremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance of the distilled models due to their dependence on priors in- cluding the amount of synthetic data used for distillation, the student architecture, training hyper-parameters and confidence of the teacher models, makes distillation a brittle compression mechanism. To mitigate this, we explore the use of post-training quantization for the com- pression of these models. Here, we find that while distillation provides gains across some low-resource languages, quantization provides more consistent performance trends for the en- tire range of languages, especially the lowest- resource languages in our target set. 1 Introduction While NLP has made giant strides in producing more accurate models, these benefits are often not transferred representatively to end-users who would eventually use a language-technology (Etha- yarajh and Jurafsky, 2020; Caselli et al., 2021). Bloated sizes, cumbersome inference times (Tao et al., 2022a) and a limited set of languages that these models serve are a few reasons for this. More specifically, their usage is hindered by access bot- tlenecks such as (a) Infrastructural Obstacles: A large percentage of end-users do not have sus- tained access to internet or high-compute devices to enjoy a stable access to cloud-inferencing of cur- rent NLP models (Ranathunga and de Silva, 2022; Diddee et al., 2022), (b) Latency Requirements: Certain NLP services (chat-bots, real-time assis- tance interfaces, etc.) require very low-inference time which requisite lightweight-models (c) Pri- vacy Constraints: The outflow of sensitive user data which is fed for inferencing to remotely hosted NLP models also has well documented issues (Sri- nath et al., 2021; Huang and Chen, 2021; Huang et al., 2020; Diddee and Kansra, 2020). Within the research that focuses on evaluating and mitigating these practical constraints, the focus on low-resource language setups has been fairly limited (Ganesh et al., 2021). For instance, while the compression of large language models has re- ceived consistent attention through analysis of prun- ing (Behnke and Heafield, 2020; Behnke et al., 2021), distillation (Bapna et al., 2022; Mghab- bar and Ratnamogan, 2020; Kim and Rush, 2016; Junczys-Dowmunt et al., 2018) and even quantiza- tion (Bondarenko et al., 2021; Zadeh et al., 2020) - much of this work …

Content truncated to 500 words