Compositional generalization with a broad-coverage semantic parser

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 44 - 54 July 14-15, 2022 ©2022 Association for Computational Linguistics Compositional generalization with a broad-coverage semantic parser Pia Weißenhorn and Lucia Donatelli and Alexander Koller Department of Language Science and Technology Saarland Informatics Campus Saarland University, Germany {piaw, donatelli, koller}@coli.uni-saarland.de Abstract We show how the AM parser, a compositional semantic parser (Groschwitz et al., 2018), can solve compositional generalization on the COGS dataset. It is the first semantic parser that achieves high accuracy on both naturally occurring language and the synthetic COGS dataset. We discuss implications for corpus and model design for learning human-like gen- eralization. Our results suggest that compo- sitional generalization can be best achieved by building compositionality into semantic parsers. 1 Introduction A growing body of recent research investigates compositional generalization, the ability of a se- mantic parser to predict the meaning of unseen sen- tences by recombining training instances in novel ways. Such generalization is thought to mimic the Principle of Compositionality (Partee, 1984), es- sential for human language learning and use. For example, COGS (Kim and Linzen, 2020), a dataset based on fragments of English, contains training in- stances with sentences semantically annotated with up to two recursive PPs; a semantic parser must then predict meaning representations for sentences with three or more recursive PPs (Table 1). Previous work has shown that compositional generalization on COGS is a difficult and com- plex task. Intricate sequence-to-sequence (seq2seq) models, which achieve very high accuracy on broad-coverage semantic parsing tasks on natu- rally occurring language (Bevilacqua et al., 2021), achieve overall accuracy of 88% or less on COGS (Akyürek and Andreas, 2021; Csordás et al., 2021; Zheng and Lapata, 2021). Much of this accuracy is due to lexical generalization, tasks that test for generalization to new words in known structures (Sec. 2); when evaluated only on structural gener- alization cases that test novel structures such as the PP example above, the accuracy of most of these models drops to 10% or less. In contrast, models that achieve high accuracy on synthetic compositional generalization datasets may not be able to generalize to naturally occur- ring language. For instance, Shaw et al. (2021) de- scribe a synchronous grammar induction approach that achieves perfect accuracy on SCAN (Lake and Baroni, 2018), but has very low accuracy on cor- pora of naturally occurring text such as GeoQuery (Zelle and Mooney, 1996) and Spider (Yu et al., 2018). Similarly, the compositional LeAR parser (Liu et al., 2021) solves COGS with near-perfect accuracy and performs very well on other synthetic datasets, but has not been evaluated on corpora of naturally occurring text. This points to a funda- mental tension between broad-coverage semantic parsing on natural text and the ability to generalize compositionally from structurally limited synthetic training sets (see also Shaw et al., 2021). To our knowledge, the only parser that does well on both is the CSL-T5 system of Qiu et al. (2022), which fine-tunes T5 using a complex data …

Content truncated to 500 words