Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4711–4717 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4711 Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings Prafull Prakash1∗, Saurabh Kumar Shashidhar1∗, Wenlong Zhao1∗, Subendhu Rongali1, Haidar Khan2, and Michael Kayser2 1University of Massachusetts Amherst 2Amazon Alexa {prafullpraka, ssaurabhkuma, wenlongzhao, srongali}@cs.umass.edu {khhaida, mikayser}@amazon.com Abstract The current state-of-the-art task-oriented se- mantic parsing models use BERT or RoBERTa as pretrained encoders; these models have huge memory footprints. This poses a chal- lenge to their deployment for voice assis- tants such as Amazon Alexa and Google As- sistant on edge devices with limited mem- ory budgets. We propose to learn compo- sitional code embeddings to greatly reduce the sizes of BERT-base and RoBERTa-base. We also apply the technique to DistilBERT, ALBERT-base, and ALBERT-large, three al- ready compressed BERT variants which at- tain similar state-of-the-art performances on semantic parsing with much smaller model sizes. We observe 95.15% ∼ 98.46% embed- ding compression rates and 20.47% ∼ 34.22% encoder compression rates, while preserving >97.5% semantic parsing performances. We provide the recipe for training and analyze the trade-off between code embedding sizes and downstream performances. 1 Introduction Conversational virtual assistants, such as Amazon Alexa, Google Home, and Apple Siri, have become increasingly popular in recent times. These systems can process queries from users and perform tasks such as playing music and finding locations. A core component in these systems is a task-oriented semantic parsing model that maps natural language expressions to structured representations contain- ing intents and slots that describe the task to per- form. For example, the expression Can you play some songs by Coldplay? may be converted to Intent: PlaySong, Artist: Coldplay, and the expres- sion Turn off the bedroom light may be converted to Intent: TurnOffLight, Device: bedroom. Task-oriented semantic parsing is traditionally approached as a joint intent classification and slot ∗Equal contribution, alphabetical order filling task. Kamath and Das (2018) provide a comprehensive survey of models proposed to solve this task. Researchers have developed semantic parsers based on Recurrent Neural Networks (Mes- nil et al., 2013; Liu and Lane, 2016; Hakkani-T¨ur et al., 2016), Convolutional Neural Networks (Xu and Sarikaya, 2013; Kim, 2014), Recursive Neural Networks (Guo et al., 2014), Capsule Networks (Sabour et al., 2017; Zhang et al., 2019), and slot- gated attention-based models (Goo et al., 2018). The current state-of-the-art models on SNIPS (Coucke et al., 2018), ATIS (Price, 1990), and Face- book TOP (Gupta et al., 2018) datasets are all based on BERT-style (Devlin et al., 2018; Liu et al., 2019) encoders and transformer architectures (Chen et al., 2019; Castellucci et al., 2019; Rongali et al., 2020). It is challenging to deploy these large models on edge devices and enable the voice assistants to op- erate locally instead of relying on central cloud services, due to the limited memory budgets on these devices. However, there has been a growing push towards the idea of TinyAI 1. In this paper, we aim to build space-efficient task-oriented …

Content truncated to 500 words