Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 108–114 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 108 Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks Jo˜ao Loula ´Ecole Polytechnique Facebook AI Research joaoloula@fb.com Marco Baroni Facebook AI Research mbaroni@fb.com Brenden M. Lake New York University Facebook AI Research brenden@nyu.edu Abstract Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it’s seen as key to the human capacity for generalization in language. Recent work (Lake and Baroni, 2018) has studied systematic compositional- ity in modern seq2seq models using general- ization to novel navigation instructions in a grounded environment as a probing tool. Lake and Baroni’s main experiment required the models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to re- combine well-trained functional words (such as “around” and “right”) in novel contexts. Our findings confirm and strengthen the ear- lier ones: seq2seq models can be impres- sively good at generalizing to novel combina- tions of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of “X around right” to “jump around right”), while failing when gen- eralization requires novel application of com- positional rules (e.g., inferring the meaning of “around right” from those of “right” and “around”). 1 Introduction Human language learning enjoys a good kind of combinatorial explosion — if a person knows the meaning of “to run” and that of “slowly”, she can immediately understand what it means “to run slowly”, even if she has never uttered or heard this expression before. This is an example of composi- tionality, the algebraic capacity to understand and produce novel combinations from known compo- nents (Montague, 1970). This principle helps to explain how, when acquiring a language, we can quickly bootstrap to a potentially infinite num- ber of expressions from very limited training data (Chomsky, 1957). Neural networks have recently been success- fully applied to many tasks requiring considerable generalization abilities (LeCun et al., 2015), in- cluding applications in the domain of natural lan- guage (Goldberg, 2017). However, it has also been observed that they require a very large num- ber of training examples to succeed, which sug- gests that they lack compositional abilities (Lake et al., 2017). There has been a substantial ear- lier debate on the extent to which neural networks display some degree of compositional generaliza- tion (e.g., Fodor and Pylyshyn, 1988; Christiansen and Chater, 1994; Marcus, 1998; Phillips, 1998; Chang, 2002; Marcus, 2003; van der Velde et al., 2004; Bowers et al., 2009; Botvinick and Plaut, 2009; Brakel and Frank, 2009; Frank, 2014). Re- cently, Lake and Baroni (2018) revisited these is- sues in light of the latest advances in deep neural networks for natural language processing. The authors introduced the SCAN dataset for studying compositionality in sequence-to- sequence (seq2seq) neural network models (Sutskever et al., 2014). …

Content truncated to 500 words