Grounding learning of modifier dynamics: An application to color naming

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1488–1493, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1488 Grounding learning of modifier dynamics: An application to color naming Xudong Han♠ Philip Schulz ♥ ♠ University of Melbourne ♥ Amazon Research xudongh1@student.unimelb.edu.au phschulz@amazon.com trevor.cohn@unimelb.edu.au Trevor Cohn ♠ Abstract Grounding is crucial for natural language un- derstanding. An important subtask is to un- derstand modified color expressions, such as “dirty blue”. We present a model of color modifiers that, compared with previous addi- tive models in RGB space, learns more com- plex transformations. In addition, we present a model that operates in the HSV color space. We show that certain adjectives are better modeled in that space. To account for all mod- ifiers, we train a hard ensemble model that se- lects a color space depending on the modifier- color pair. Experimental results show signif- icant and consistent improvements compared to the state-of-the-art baseline model.1 1 Introduction Grounded color descriptions are employed to de- scribe colors which are not covered by basic color terms (Monroe et al., 2017). For instance, “green- ish blue” cannot be expressed by only “blue” or “green”. Grounded learning of modifiers, as a result, is essential for grounded language understanding problems such as image caption- ing (Karpathy and Fei-Fei, 2015), visual question answering (Goyal et al., 2017) and object recogni- tion (van de Sande et al., 2010). In this paper, we present models that are able to predict the RGB code of a target color given a reference color and a modifier. For exam- ple, as shown in Figure 1, given a reference color code ⃗r = � 101 55 0 �⊤ and a modifier m = “greenish”, our models are trained to pre- dict the target color code ⃗t = � 105 97 18 �⊤. The state-of-the-art approach for this task (Winn and Muresan, 2018) represents both colors and modifiers as vectors in RGB space, and learns a 1Code available at https://github.com/ HanXudong/GLoM Figure 1: Examples of the grounded modifier mod- elling task, shown in RGB space. Given the reference and modifier, the system must predict the target color. vector representation of modifiers ⃗m as part of a simple additive model, ⃗r + ⃗m ≈ ⃗r, in RGB color space. For instance, given the reference color ⃗r = � 229 0 0 �⊤, the target color ⃗t = � 132 0 0 �⊤, the modifier m = “darker” is learned as a vector ⃗m = � −97 0 0 �⊤. This model works well when the modifier is well repre- sented as a single vector independent of the ref- erence color,2 but fails to model modifiers with more complex transformations, for example, color related modifiers, like “greenish”, which are bet- ter modelled through color interpolation. To fit a better model, we assume that there are approximate intersection points for the extension lines of modifier vectors, for instance, Figure 2c …

Content truncated to 500 words