Disentangled Sequence to Sequence Learning for Compositional Generalization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4256 - 4268 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Disentangled Sequence to Sequence Learning for Compositional Generalization Hao Zheng and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB Hao.Zheng@ed.ac.uk mlap@inf.ed.ac.uk Abstract There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen composi- tions of seen components. We demonstrate that one of the reasons hindering compositional gen- eralization relates to representations being en- tangled. We propose an extension to sequence- to-sequence models which encourages disen- tanglement by adaptively re-encoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than cap- turing it all in a single forward pass. Experi- mental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization. 1 1 Introduction When humans use language, they exhibit composi- tional generalization; they are able to produce and understand a potentially infinite number of novel linguistic expressions by systematically combining known atomic components (Chomsky, 2014; Mon- tague, 1970). For example, if a person knows the meaning of the utterance “A boy ate the cake on the table in a house” and the verb “like”, it is natural for them to understand the utterance “A boy likes the cake on the table in a house” when they encounter it for the first time (see Table 1). Humans are also adept at recognizing novel combinations of famil- iar syntactic structure, e.g., they would have no trouble processing the above sentence if the prepo- sition “beside the tree” were added to it, despite not having previously seen the phrase “in a house beside the tree” (see Table 1). 1Our code is available at https://github.com/ mswellhao/Dangle. Training Set A boy ate the cake on the table in a house. *cake(x4); *table(x7); boy(x1) AND eat.agent(x2, x1) AND eat.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) Test Set (Lexical Generalization) A boy likes the cake on the table in a house. *cake(x4); *table(x7); boy(x1) AND like.agent(x2, x1) AND like.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) Test Set (Structural Generalization) A boy ate the cake on the table in a house beside the tree. *cake(x4); *table(x7); *tree(x13); boy(x1) AND eat.agent(x2, x1) AND eat.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) AND house.nmod.beside(x10, x13) Table 1: Examples from COGS (Kim and Linzen, 2020) showcasing lexical and structural generalization. In lexical generalization, a familiar word (e.g., like) is at- tested in a familiar syntactic structure but the resulting combination has not been seen before. In structural generalization, familiar syntactic components give rise to novel combinations (e.g., only prepositional phrases with nesting depth 2 have been previously …

Content truncated to 500 words