Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 922–938 August 1–6, 2021. ©2021 Association for Computational Linguistics 922 Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? Peter Shaw Ming-Wei Chang Panupong Pasupat Kristina Toutanova Google Research {petershaw,mingweichang,ppasupat,kristout}@google.com Abstract Sequence-to-sequence models excel at han- dling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has mo- tivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural lan- guage variation and compositional generaliza- tion? To better assess this capability, we pro- pose new train and test splits of non-synthetic datasets. We demonstrate that strong exist- ing approaches do not perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a high- precision grammar-based approach with a pre- trained sequence-to-sequence model. It outper- forms existing approaches across several com- positional generalization challenges on non- synthetic data, while also being competitive with the state-of-the-art on standard evalua- tions. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing. 1 Introduction Sequence-to-sequence (seq2seq) models have been widely used in semantic parsing (Dong and Lap- ata, 2016; Jia and Liang, 2016) and excel at han- dling the natural language variation1 of human- generated queries. However, evaluations on syn- thetic2 tasks such as SCAN (Lake and Baroni, 1We use the term natural language variation in a broad sense to refer to the many different ways humans can express the same meaning in natural language, including differences in word choice and syntactic constructions. 2We make a coarse distinction between synthetic datasets, where natural language utterances are generated by a program, Specialized architectures with strong compositional bias Under-explored General-purpose pre-trained models (e.g. seq2seq) PREDOMINANT APPROACHES SYNTHETIC NON-SYNTHETIC NATURAL LANGUAGE VARIATION COMPOSITIONAL GENERALIZATION Figure 1: We study whether a semantic parsing ap- proach can handle both out-of-distribution composi- tional generalization and natural language variation. Existing approaches are commonly evaluated across only one dimension. 2018) have shown that seq2seq models often gener- alize poorly to out-of-distribution compositional ut- terances, such as “jump twice” when only “jump”, “walk”, and “walk twice” are seen during training. This ability to generalize to novel combinations of the elements observed during training is referred to as compositional generalization. This has motivated many specialized architec- tures that improve peformance on SCAN (Li et al., 2019; Russin et al., 2019; Gordon et al., 2019; Lake, 2019; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020). However, most approaches have only been evaluated on synthetic datasets. While synthetic datasets enable precise, interpretable evaluation of specific phenomena, they are less …

Content truncated to 500 words