Structural generalization is hard for sequence-to-sequence models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048 - 5062 December 7-11, 2022 ©2022 Association for Computational Linguistics Structural generalization is hard for sequence-to-sequence models Yuekun Yao and Alexander Koller Department of Language Science and Technology Saarland Informatics Campus Saarland University, Saarbrücken, Germany {ykyao, koller}@coli.uni-saarland.de Abstract Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, in- cluding ones that require predicting linguistic structure. However, recent work on composi- tional generalization has shown that seq2seq models achieve very low accuracy in general- izing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the rea- sons for these limitations. 1 Introduction Humans are able to understand and produce lin- guistic structures they have never observed before (Chomsky, 1957; Fodor and Pylyshyn, 1988; Fodor and Lepore, 2002). From limited, finite observa- tions, they generalize at an early age to an infinite variety of novel structures using recursion. They can also assign meaning to these, using the Princi- ple of Compositionality. This ability to generalize to unseen structures is important for NLP systems in low-resource settings, such as underresourced languages or projects with a limited annotation bud- get, where a user can easily use structures that had no annotations in training. Over the past few years, large pretrained sequence-to-sequence (seq2seq) models, such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), have brought tremendous progress to many NLP tasks. This includes linguistically complex tasks such as broad-coverage semantic parsing, where e.g. a lightly modified BART set a new state of the art on AMR parsing (Bevilacqua et al., 2021). However, there have been some concerns that seq2seq models may have difficulties with com- positional generalization, a class of tasks in seman- tic parsing where the training data is structurally impoverished in comparison to the test data (Lake and Baroni, 2018; Keysers et al., 2020). We focus on the COGS dataset of Kim and Linzen (2020) be- cause some of its generalization types specifically target structural generalization, i.e. the ability to generalize to unseen structures. In this paper, we make two contributions. First, we offer evidence that structural generalization is systematically hard for seq2seq models. On the semantic parsing task of COGS, seq2seq mod- els don’t fail on compositional generalization as a whole, but specifically on the three COGS gener- alization types that require generalizing to unseen linguistic structures, achieving accuracies below 10%. This is true both for BART and T5 and for seq2seq models that were specifically developed for COGS. What’s more, BART and T5 fail simi- larly on syntax and even POS tagging variants of COGS (introduced in this paper), indicating that they do not …

Content truncated to 500 words