Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9157 - 9179 December 7-11, 2022 ©2022 Association for Computational Linguistics Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing Linlu Qiu1∗ Peter Shaw2 Panupong Pasupat2 Tianze Shi2 Jonathan Herzig2 Emily Pitler2 Fei Sha2 Kristina Toutanova2 1Massachusetts Institute of Technology 2Google Research linluqiu@mit.edu,{petershaw,ppasupat,tianze,jherzig,epitler,fsha,kristout}@google.com Abstract Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution com- positional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve composi- tional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for apply- ing a pre-trained language model to a new task: fine-tuning all parameters, prompt tun- ing, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution composi- tional generalization in semantic parsing evalu- ations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further po- tential improvements from scaling as it ex- hibits a more positive scaling curve. Addition- ally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compo- sitional generalization, while our analysis also suggests promising directions for future work. 1 Introduction Compositional generalization is the ability to gener- alize to novel combinations of previously observed elements. For example, we may ask a model to interpret “she loves the dog” when “she”, “loves”, and “the dog” were seen separately but not in com- bination with each other during training. Improv- ing compositional generalization is believed to be ∗Work done as part of the Google AI Residency program. important for approaching human-like language understanding (Lake et al., 2017; Battaglia et al., 2018). In addition, models that are deployed for real-world applications often need to generalize to new compositions of elements not well-represented in static and often biased annotated training sets (Herzig and Berant, 2019; Yin et al., 2021). In this paper we focus on compositional generalization for semantic parsing, the task of mapping utterances to logical forms with precisely defined semantics. Despite their strong performance on many tasks, pre-trained language models1 (LMs) such as T5 (Raffel et al., 2020) have been shown to struggle on compositional generalization (Lake and Baroni, 2018; Furrer et al., 2020; Shaw et al., 2021). How- ever, recent work has shown considerable improve- ments across a range of NLP tasks from scaling up model size (Brown et al., 2020; Chowdhery et al., 2022). Can scaling up the number of parameters of pre-trained language models also improve com- positional generalization in semantic parsing? Understanding the relationship between …

Content truncated to 500 words