Learning to Substitute Spans towards Improving Compositional Generalization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2791–2811 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning to Substitute Spans towards Improving Compositional Generalization Zhaoyi Li1,2, Ying Wei3∗and Defu Lian1,2∗ 1School of Computer Science and Technology, University of Science and Technology of China 2State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China 3Department of Computer Science, City University of Hong Kong lizhaoyi777@mail.ustc.edu.cn, yingwei@cityu.edu.hk, liandefu@ustc.edu.cn Abstract Despite the rising prevalence of neural se- quence models, recent empirical evidences sug- gest their deficiency in compositional general- ization. One of the current de-facto solutions to this problem is compositional data augmenta- tion, aiming to incur additional compositional inductive bias. Nonetheless, the improvement offered by existing handcrafted augmentation strategies is limited when successful system- atic generalization of neural sequence models requires multi-grained compositional bias (i.e., not limited to either lexical or structural biases only) or differentiation of training sequences in an imbalanced difficulty distribution. To address the two challenges, we first propose a novel compositional augmentation strategy dubbed Span Substitution (SpanSub) that en- ables multi-grained composition of substantial substructures in the whole training set. Over and above that, we introduce the Learning to Substitute Span (L2S2) framework which em- powers the learning of span substitution proba- bilities in SpanSub in an end-to-end manner by maximizing the loss of neural sequence models, so as to outweigh those challenging composi- tions with elusive concepts and novel surround- ings. Our empirical results on three standard compositional generalization benchmarks, in- cluding SCAN, COGS and GeoQuery (with an improvement of at most 66.5%, 10.3%, 1.2%, respectively), demonstrate the superiority of SpanSub, L2S2 and their combination. 1 Introduction The secret for human beings to learning so quickly with little supervision has been demonstrated to be associated with the powerful ability of systematic generalization, being capable of producing an in- finite number of novel combinations on the basis of known components (Chomsky, 1957). In stark contrast, a large body of recent evidence suggests that current state-of-the-art neural sequence models ∗Corresponding authors N: Emma V: saw N: a cat N: A hedgehog V: met N: Paula V: saw N: a cat N: Paula A hedgehog met Paula . Emma saw a cat . Paula saw a cat . (a) word-level composition N: James V: knew N: a cat V: was scared N: Liam V: was told N: the cake V: was ate N: Liam V: was told N: James V: knew N: a cat V: was scared James knew that a cat was scared . Liam was told that the cake was ate . Liam was told that James knew that a cat was scared . (b) subtree-level composition N: The rat V: scared N: Mary V: scared N: Mary N: a cat PREP: on N: the mat N: Emma V: saw N: a cat PREP: on N: the mat PREP: beside N: the tree Emma saw a cat on the mat beside the tree . The rat scared Mary . A …

Content truncated to 500 words