LexSym: Compositionality as Lexical Symmetry

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 639–657 July 9-14, 2023 ©2023 Association for Computational Linguistics LexSym: Compositionality as Lexical Symmetry Ekin Akyürek Jacob Andreas Massachusetts Institute of Technology {akyurek,jda}@mit.edu Abstract In tasks like semantic parsing, instruction fol- lowing, and question answering, standard deep networks fail to generalize compositionally from small datasets. Many existing approaches overcome this limitation with model archi- tectures that enforce a compositional process of sentence interpretation. In this paper, we present a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. Informally, we prove that whenever a task can be solved by a compositional model, there is a corresponding data augmentation scheme—a procedure for transforming exam- ples into other well-formed examples—that imparts compositional inductive bias on any model trained to solve the same task. We de- scribe a procedure called LEXSYM that discov- ers these transformations automatically, then applies them to training data for ordinary neu- ral sequence models. Unlike existing composi- tional data augmentation procedures, LEXSYM can be deployed agnostically across text, struc- tured data, and even images. It matches or sur- passes state-of-the-art, task-specific models on COGS semantic parsing, SCAN and ALCHEMY instruction following, and CLEVR-COGENT visual question answering datasets. 1 Introduction A central challenge in natural language processing is the design of models and learning algorithms that are simultaneously flexible enough to capture the variability of human language and structured enough to generalize in predictable and human- like ways. One important source of structure is the principle of compositionality, which (in one formulation) states that sentence meanings can be computed from a lexicon of word meanings and a set of composition rules governing how meanings combine (Montague, 1970b). A long line of lan- guage processing research has operationalized the principle of compositionality as a constraint on model architectures, via independence assump- tions or parameter tying schemes that ensure a com- positional process of sentence interpretation (Lewis and Stearns, 1968; Andreas et al., 2016). Composi- tional models enjoy sample-efficient learning and strong generalization in tasks from machine trans- lation to question answering (McCoy et al., 2020). But much of human language is not (or at least not straightforwardly) compositional. Idioms, dis- fluencies, and context-sensitive meanings present major challenges to models in which all predic- tions must derive from a sequence of local com- position operations. In recent years, more generic model architectures such as recurrent neural net- works (RNNs) and transformers, with no explicit compositional scaffolding, have consistently out- performed compositional models in language pro- cessing tasks with natural data (Wu et al., 2016). However, these models capture linguistic regular- ities only when trained on enormous amounts of data, and make surprising or problematic predic- tions when presented with novel word collocations or syntactic structures (Lake and Baroni, 2018). How can we train unstructured neural sequence models that generalize compositionally? Recent work has introduced several compositional data augmentation schemes: rule-based procedures or learned models that synthesize …

Content truncated to 500 words