Model Compression for Domain Adaptation through Causal Effect Estimation

Model Compression for Domain Adaptation through Causal Effect Estimation Guy Rotman∗, Amir Feder∗, Roi Reichart Faculty of Industrial Engineering and Management, Technion, IIT, Israel grotman@campus.technion.ac.il feder@campus.technion.ac.il roiri@technion.ac.il Abstract Recent improvements in the predictive quality of natural language processing systems are of- ten dependent on a substantial increase in the number of model parameters. This has led to various attempts of compressing such models, but existing methods have not considered the differences in the predictive power of various model components or in the generalizability of the compressed models. To understand the connection between model compression and out-of-distribution generalization, we define the task of compressing language representa- tion models such that they perform best in a domain adaptation setting. We choose to ad- dress this problem from a causal perspective, attempting to estimate the average treatment effect (ATE) of a model component, such as a single layer, on the model’s predictions. Our proposed ATE-guided Model Compres- sion scheme (AMoC), generates many model candidates, differing by the model components that were removed. Then, we select the best candidate through a stepwise regression model that utilizes the ATE to predict the expected performance on the target domain. AMoC outperforms strong baselines on dozens of do- main pairs across three text classification and sequence tagging tasks.1 1 Introduction The rise of deep neural networks has transformed the way we represent language, allowing models to learn useful features directly from raw inputs. However, recent improvements in the predictive quality of language representations are often re- lated to a substantial increase in the number of model parameters. Indeed, the introduction of the Transformer architecture (Vaswani et al., 2017) ∗Authors contributed equally. 1Our code and data are available at: https://github .com/rotmanguy/AMoC. and attention-based models (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) have improved performance on most natural language processing (NLP) tasks, while facilitating a large increase in model sizes. Since large models require a significant amount of computation and memory during training and inference, there is a growing demand for com- pressing such models while retaining the most relevant information. While recent attempts have shown promising results (Sanh et al., 2019), they have some limitations. Specifically, they attempt to mimic the behavior of the larger models without trying to understand the information preserved or lost in the compression process. In compressing the information represented in billions of parameters, we identify three main challenges. First, current methods for model compression are not interpretable. While the im- portance of different model parameters is certainly not uniform, it is hard to know a priori which of the model components should be discarded in the compression process. This notion of fea- ture importance has not yet trickled down into compression methods, and they often attempt to solve a dimensionality reduction problem where a smaller model aims to mimic the predictions of the larger model. Nonetheless, not all parameters are born equal, and only a subset of the informa- tion captured in the network is actually useful for …

Content truncated to 500 words