Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding Will Monroe,1 Robert X.D. Hawkins,2 Noah D. Goodman,1,2 and Christopher Potts3 Departments of 1Computer Science, 2Psychology, and 3Linguistics Stanford University, Stanford, CA 94305 wmonroe4@cs.stanford.edu, {rxdh, ngoodman, cgpotts}@stanford.edu Abstract We present a model of pragmatic referring expression interpretation in a grounded com- munication task (identifying colors from de- scriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recur- sive pragmatic reasoning framework. Exper- iments show that this combined pragmatic model interprets color descriptions more ac- curately than the classifiers from which it is built, and that much of this improvement re- sults from combining the speaker and listener perspectives. We observe that pragmatic rea- soning helps primarily in the hardest cases: when the model must distinguish very simi- lar colors, or when few utterances adequately express the target color. Our findings make use of a newly-collected corpus of human ut- terances in color reference games, which ex- hibit a variety of pragmatic behaviors. We also show that the embedded speaker model repro- duces many of these pragmatic behaviors. 1 Introduction Human communication is situated. In using lan- guage, we are sensitive to context and our interlocu- tors’ expectations, both when choosing our utter- ances (as speakers) and when interpreting the utter- ances we hear (as listeners). Visual referring tasks exercise this complex process of grounding, in the environment and in our mental models of each other, and thus provide a valuable test-bed for computa- tional models of production and comprehension. Table 1 illustrates the situated nature of reference understanding with descriptions of colors from a Context Utterance 1. xxxx xxxx xxxx darker blue 2. xxxx xxxx xxxx Purple 3. xxxx xxxx xxxx blue 4. xxxx xxxx xxxx blue Table 1: Examples of color reference in context, taken from our corpus. The target color is boxed. The speaker’s description is shaped not only by this target, but also by the other context colors and their relationships. task-oriented dialogue corpus we introduce in this paper. In these dialogues, the speaker is trying to identify their (privately assigned) target color for the listener. In context 1, the comparative darker implic- itly refers to both the target (boxed) and one of the other colors. In contexts 2 and 3, the target color is the same, but the distractors led the speaker to choose different basic color terms. In context 4, blue is a pragmatic choice even though two colors are shades of blue, because the interlocutors assume about each other that they find the target color a more prototypical representative of blue and would prefer other descriptions (teal, cyan) for the middle color. The fact that blue appears in three of these four cases highlights the flexibility and context de- pendence of color descriptions. In this paper, we present a scalable, learned model of pragmatic language understanding. The model is built around a version of the Rational Speech Acts (RSA) model (Frank and Goodman, …

Content truncated to 500 words