Recursive Routing Networks: Learning to Compose Modules for Language Understanding

Proceedings of NAACL-HLT 2019, pages 3631–3648 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3631 Recursive Routing Networks: Learning to Compose Modules for Language Understanding Ignacio Cases1, Clemens Rosenbaum2, Matthew Riemer3, Atticus Geiger1, Tim Klinger3, Alex Tamkin1, Olivia Li1, Sandhini Agarwal1, Joshua D. Greene4, Dan Jurafsky1, Christopher Potts1, Lauri Karttunen1 1Stanford University 2University of Massachusetts Amherst 3IBM Research 4Harvard University cases@stanford.edu, cgbr@cs.umass.edu Abstract We introduce Recursive Routing Networks (RRNs), which are modular, adaptable mod- els that learn effectively in diverse environ- ments. RRNs consist of a set of functions, typ- ically organized into a grid, and a meta-learner decision-making component called the router. The model jointly optimizes the parameters of the functions and the meta-learner’s pol- icy for routing inputs through those functions. RRNs can be incorporated into existing ar- chitectures in a number of ways; we explore adding them to word representation layers, re- current network hidden layers, and classifier layers. Our evaluation task is natural language inference (NLI). Using the MULTINLI corpus, we show that an RRN’s routing decisions re- flect the high-level genre structure of that cor- pus. To show that RRNs can learn to specialize to more fine-grained semantic distinctions, we introduce a new corpus of NLI examples in- volving implicative predicates, and show that the model components become fine-tuned to the inferential signatures that are characteris- tic of these predicates. 1 Introduction Human cognition has an extraordinary ability to modularize, decomposing problems and solving them by re-composing elements from prior solu- tions, and this ability is nowhere more evident than in language understanding (Partee, 1984; Janssen, 1997). Most machine learning architectures lack this modularity, which limits their ability to gener- alize and leaves them susceptible to catastrophic interference (McCloskey and Cohen, 1989) – for- getting past skills when acquiring new ones. We propose to address this need for modularity by applying Routing Networks (Rosenbaum et al., 2017) to natural language understanding. Routing Networks are self-organizing networks with two components (Figure 1): a set of function blocks Router 5 Recursive Routing Network 3 2 4 1 6 Figure 1: Given a premise–hypothesis pair x, e.g., t man- aged to do s and t did s (1), the model needs to learn to pre- dict entails (6). The router (2) estimates the value of applying each of the available sub-functions to the input. Given that manage has certain semantic properties, the router may se- lect (3) a function f1 specialized to them. The module is then applied (4), yielding f1(x). This process repeats (5), now us- ing f1(x), selecting and applying another sub-function, and so on, until the router is confident in its prediction (6). which can be applied to transform the input, and a router which makes decisions about which func- tion block to apply next. Here we introduce Re- cursive Routing Networks (RRNs), in which there is a single set of composable functions, recursively chosen by the router. RRNs can be applied to different components of modern language under- standing …

Content truncated to 500 words