Reasoning about Pragmatics with Neural Listeners and Speakers

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Reasoning about Pragmatics with Neural Listeners and Speakers Jacob Andreas and Dan Klein Computer Science Division University of California, Berkeley {jda,klein}@cs.berkeley.edu Abstract We present a model for contrastively describ- ing scenes, in which context-specific behav- ior results from a combination of inference- driven pragmatics and learned semantics. Like previous learned approaches to language gen- eration, our model uses a simple feature- driven architecture (here a pair of neural “lis- tener” and “speaker” models) to ground lan- guage in the world. Like inference-driven ap- proaches to pragmatics, our model actively reasons about listener behavior when select- ing utterances. For training, our approach re- quires only ordinary captions, annotated with- out demonstration of the pragmatic behavior the model ultimately exhibits. In human eval- uations on a referring expression game, our approach succeeds 81% of the time, compared to 69% using existing techniques. 1 Introduction We present a model for describing scenes and ob- jects by reasoning about context and listener behav- ior. By incorporating standard neural modules for image retrieval and language modeling into a prob- abilistic framework for pragmatics, our model gen- erates rich, contextually appropriate descriptions of structured world representations. This paper focuses on a reference game RG played between a listener L and a speaker S. 1. Reference candidates r1 and r2 are re- vealed to both players. 2. S is secretly assigned a random target t ∈ {1, 2}. 3. S produces a description d = S(t, r1, r2), which is shown to L. 4. L chooses c = L(d, r1, r2). 5. Both players win if c = t. (RG) (a) target (b) distractor the owl is sitting in the tree (c) description Figure 1: Sample output from our model. When presented with a target image (a) in contrast with a distractor image (b), the model generates a description (c). This description mentions a tree, the distinguishing object present in (a) but not in (b), and situates it with respect to other objects and events in the scene. Figure 1 shows an example drawn from a standard captioning dataset (Zitnick et al., 2014). In order for the players to win, S’s description d must be pragmatic: it must be informative, fluent, concise, and must ultimately encode an understand- ing of L’s behavior. In Figure 1, for example, the owl is wearing a hat and the owl is sitting in the tree are both accurate descriptions of the target image, but only the second allows a human listener to suc- ceed with high probability. RG is the focus of many papers in the computational pragmatics literature: it provides a concrete generation task while eliciting a broad range of pragmatic behaviors, including con- versational implicature (Benotti and Traum, 2009) and context dependence (Smith et al., 2013). Exist- ing computational models of pragmatics can be di- vided into two broad lines of work, which we term …

Content truncated to 500 words