Neural Event Semantics for Grounded Language Understanding

Neural Event Semantics for Grounded Language Understanding Shyamal Buch Li Fei-Fei Noah D. Goodman Stanford University, United States {shyamal,feifeili}@cs.stanford.edu ngoodman@stanford.edu Abstract We present a new conjunctivist framework, neural event semantics (NES), for composi- tional grounded language understanding. Our approach treats all words as classifiers that compose to form a sentence meaning by mul- tiplying output scores. These classifiers apply to spatial regions (events) and NES derives its semantic structure from language by rout- ing events to different classifier argument inputs via soft attention. NES is trainable end-to-end by gradient descent with min- imal supervision. We evaluate our method on compositional grounded language tasks in controlled synthetic and real-world settings. NES offers stronger generalization capability than standard function-based compositional frameworks, while improving accuracy over state-of-the-art neural methods on real-world language tasks. 1 Introduction Capturing the compositional semantics of grounded language is a long-standing goal in natural language processing. Composition yields systematicity, and is thus essential to developing systems that can generalize broadly in real-world settings. Recent progress with neural module net- works (Andreas et al., 2016b; Hu et al., 2017) and related models (Johnson et al., 2017b; Yi et al., 2018; Bahdanau et al., 2019a) have moved neural network methods closer to this goal. These works are largely based on the idea, functionism (Montague, 1970), that semantic com- position is function composition. In Figure 1(a), function predicates compose by nesting: Predi- cates like ‘‘red’’ and ‘‘circle’’ operate on sets of elements, progressively filtering them at each Project Website: https://neural-event-semantics.github.io/. step (circle(red(x))). The final relational predicate above is thus several steps removed from the original inputs x,y. Similarly, in mod- ule networks, atomic module blocks compose by sequentially passing outputs of intermediate blocks to later modules. The diverse composition ruleset needed to coordinate function inputs and outputs leads to complexity in this paradigm, which has practical implications for its fundamen- tal learnability. Indeed, neural module network instantiations of this framework often depend on low-level ground truth module layout programs (Johnson et al., 2017b) or large amounts of training data to sustain end-to-end reinforcement learning methods (Yi et al., 2018; Mao et al., 2019). While functionism is the dominant paradigm in linguistic semantics, there is an intriguing alterna- tive: event semantics (Davidson, 1967). Conjunc- tivism (Pietroski, 2005) is a particularly powerful version of event semantics, wherein the only com- position operator is conjunction—structure arises by routing event variables to the function pred- icates. We illustrate this key difference between paradigms in Figure 1: in a conjunctivist set- ting (Figure 1(b)), even the relational above has events e1, e2 directly routed as input, rather than taking inputs that are output results of a sequence of filter operations. Overall meaning is still preserved, since e1 concurrently routes to (red, circle) and e2 to (green, square). All mod- ule outputs directly contribute to the final truth value without intermediate steps. Altogether, this shift from deriving compositional structure by functional module layout to conjunctive events routing offers a path to improved learnability; we explore the implications of …

Content truncated to 500 words