Compositional Generalisation with Structured Reordering and Fertility Layers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2172–2186 May 2-6, 2023 ©2023 Association for Computational Linguistics Compositional Generalisation with Structured Reordering and Fertility Layers Matthias Lindemann1 and Alexander Koller2 and Ivan Titov1,3 1 ILCC, University of Edinburgh, 2 LST, Saarland University, 3 ILLC, University of Amsterdam m.m.lindemann@sms.ed.ac.uk, koller@coli.uni-saarland.de, ititov@inf.ed.ac.uk Abstract Seq2seq models have been shown to strug- gle with compositional generalisation, i.e. gen- eralising to new and potentially more com- plex structures than seen during training. Tak- ing inspiration from grammar-based models that excel at compositional generalisation, we present a flexible end-to-end differentiable neu- ral model that composes two structural opera- tions: a fertility step, which we introduce in this work, and a reordering step based on previous work (Wang et al., 2021). To ensure differentia- bility, we use the expected value of each step, which we compute using dynamic program- ming. Our model outperforms seq2seq models by a wide margin on challenging compositional splits of realistic semantic parsing tasks that re- quire generalisation to longer examples. It also compares favourably to other models targeting compositional generalisation.1 1 Introduction Many NLP tasks require translating an input object such as a sentence into a structured output object such as a semantic parse. Recently, these tasks have been approached with seq2seq models with great success. However, seq2seq models also have been shown to struggle out-of-distribution on com- positional generalisation (Lake and Baroni, 2018; Finegan-Dollak et al., 2018; Kim and Linzen, 2020; Hupkes et al., 2020), i.e. the model fails on exam- ples that contain unseen compositions or deeper recursion of phenomena that it handles correctly in isolation. Consider the example in Fig. 1. Arguably, any model that produces the given semantic parse from the input in a generalisable way has to capture the correspondence between fragments of the input and fragments of the output, at least implicitly. This is challenging because of structural mismatches 1https://github.com/namednil/f-then-r Figure 1: We model structural seq2seq tasks as the com- position of differentiable fertility and phrase reordering layers. The model is trained end-to-end without direct supervision of the two structural layers. between input and output. For example, the frag- ment contributed by “flights" is discontinuous and intertwined with the rest of the semantic parse. In contrast to seq2seq models, grammar-based models such as synchronous context-free grammars (SCFGs) (Lewis and Stearns, 1968; Chiang, 2007) explicitly capture the compositional process behind the data and therefore perform very well in compo- sitional generalisation setups. They can also model common structural mismatches like the one shown in the example. However, grammar-based models are rigid and brittle and thus do not scale well. In this work, we take inspiration from grammar- based models and present an end-to-end differen- tiable neural model that is both flexible and gen- eralises well compositionally. Our model consists of two structural layers: a phrase reordering layer originally introduced by Wang et al. (2021) and a fertility layer, new in this work, which creates zero or more copies of …

Content truncated to 500 words