Compositional Generalization for Primitive Substitutions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4293–4302, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4293 Compositional Generalization for Primitive Substitutions Yuanpeng Li, Liang Zhao, Jianyu Wang, Joel Hestness Baidu Research {yuanpeng16,jthestness}@gmail.com {zhaoliang07,wangjianyu02}@baidu.com Abstract Compositional generalization is a basic mech- anism in human language learning, but current neural networks lack such ability. In this paper, we conduct fundamental research for encod- ing compositionality in neural networks. Con- ventional methods use a single representation for the input sentence, making it hard to ap- ply prior knowledge of compositionality. In contrast, our approach leverages such knowl- edge with two representations, one generat- ing attention maps, and the other mapping at- tended input words to output symbols. We re- duce the entropy in each representation to im- prove generalization. Our experiments demon- strate significant improvements over the con- ventional methods in five NLP tasks includ- ing instruction learning and machine transla- tion. In the SCAN domain, it boosts accura- cies from 14.0% to 98.8% in Jump task, and from 92.0% to 99.7% in TurnLeft task. It also beats human performance on a few-shot learn- ing task. We hope the proposed approach can help ease future research towards human-level compositional language learning. Source code is available online1. 1 Introduction Humans learn language in a flexible and efficient way by leveraging systematic compositionality, the algebraic capacity to understand and produce large amount of novel combinations from known com- ponents (Chomsky, 1957; Montague, 1970). For example, if a person knows how to “step”, “step twice” and “jump”, then it is natural for the per- son to know how to “jump twice” (Figure 1). This compositional generalization is critical in human cognition (Minsky, 1986; Lake et al., 2017), and it helps humans learn language from a limited amount of data, and extend to unseen sentences. 1https://github.com/yli1/CGPS Figure 1: Problem illustration. If a person knows how to “step”, “step twice” and “jump”, then the person should know how to “jump twice”. Although this is straightforward for humans, it remains an open prob- lem to equip machines with such ability. In recent years, deep neural networks have made many breakthroughs in various problems (LeCun et al., 2015; Krizhevsky et al., 2012; Yu and Deng, 2012; He et al., 2016; Wu and et al, 2016). How- ever, there have been critiques that neural net- works do not have compositional generalization ability (Fodor and Pylyshyn, 1988; Marcus, 1998; Fodor and Lepore, 2002; Marcus, 2003; Calvo and Symons, 2014). Our observation is that conventional methods use a single representation for the input sentence, which makes it hard to apply prior knowledge of compositionality. In contrast, our approach lever- ages such knowledge with two representations, one generating attention maps, and the other mapping attended input words to output symbols. We re- duce each of their entropies to improve general- ization. This mechanism equips the proposed ap- proach with the ability to understand and …

Content truncated to 500 words