Learning to Generate Compositional Color Descriptions

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2243–2248, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning to Generate Compositional Color Descriptions Will Monroe,1 Noah D. Goodman,2 and Christopher Potts3 Departments of 1Computer Science, 2Psychology, and 3Linguistics Stanford University, Stanford, CA 94305 wmonroe4@cs.stanford.edu, {ngoodman, cgpotts}@stanford.edu Abstract The production of color language is essential for grounded language generation. Color de- scriptions have many challenging properties: they can be vague, compositionally complex, and denotationally rich. We present an effec- tive approach to generating color descriptions using recurrent neural networks and a Fourier- transformed color representation. Our model outperforms previous work on a conditional language modeling task over a large corpus of naturalistic color descriptions. In addition, probing the model’s output reveals that it can accurately produce not only basic color terms but also descriptors with non-convex denota- tions (“greenish”), bare modifiers (“bright”, “dull”), and compositional phrases (“faded teal”) not seen in training. 1 Introduction Color descriptions represent a microcosm of grounded language semantics. Basic color terms like “red” and “blue” provide a rich set of seman- tic building blocks in a continuous meaning space; in addition, people employ compositional color de- scriptions to express meanings not covered by ba- sic terms, such as “greenish blue” or “the color of the rust on my aunt’s old Chevrolet” (Berlin and Kay, 1991). The production of color language is essential for referring expression generation (Krah- mer and Van Deemter, 2012) and image captioning (Kulkarni et al., 2011; Mitchell et al., 2012), among other grounded language generation problems. We consider color description generation as a grounded language modeling problem. We present Color Top-1 Sample (83, 80, 28) “green” “very green” (232, 43, 37) “blue” “royal indigo” (63, 44, 60) “olive” “pale army green” (39, 83, 52) “orange” “macaroni” Table 1: A selection of color descriptions sampled from our model that were not seen in training. Color triples are in HSL. Top-1 shows the model’s highest-probability prediction. an effective new model for this task that uses a long short-term memory (LSTM) recurrent neural net- work (Hochreiter and Schmidhuber, 1997; Graves, 2013) and a Fourier-basis color representation in- spired by feature representations in computer vision. We compare our model with LUX (McMahan and Stone, 2015), a Bayesian generative model of color semantics. Our model improves on their approach in several respects, which we demonstrate by exam- ining the meanings it assigns to various unusual de- scriptions: (1) it can generate compositional color descriptions not observed in training (Fig. 3); (2) it learns correct denotations for underspecified modi- fiers, which name a variety of colors (“dark”, “dull”; Fig. 2); and (3) it can model non-convex denota- tions, such as that of “greenish”, which includes both greenish yellows and blues (Fig. 4). As a result, our model also produces significant improvements on several grounded language modeling metrics. 2 Model formulation Formally, a model of color description generation is a probability distribution S(d | c) over sequences of 2243 c <s> light blue …

Content truncated to 500 words