Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6536–6542 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6536 Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms Simeon Sch¨uz Friedrich Schiller University Jena simeon.schuez@uni-jena.de Sina Zarrieß Friedrich Schiller University Jena sina.zarriess@uni-jena.de Abstract In human cognition, world knowledge sup- ports the perception of object colours: know- ing that trees are typically green helps to per- ceive their colour in certain contexts. We go beyond previous studies on colour terms us- ing isolated colour swatches and study visual grounding of colour terms in realistic objects. Our models integrate processing of visual in- formation and object-specific knowledge via hard-coded (late) or learned (early) fusion. We find that both models consistently outperform a bottom-up baseline that predicts colour terms solely from visual inputs, but show interesting differences when predicting atypical colours of so-called colour diagnostic objects. Our models also achieve promising results when tested on new object categories not seen dur- ing training. 1 Introduction Research on human perception has shown that world knowledge supports the processing of sen- sory information (Mitterer et al., 2009; Ishizu, 2013). For instance, humans have been found to use their knowledge about typical colours of an object when perceiving an instance of that ob- ject, in order to compensate for, e.g., perceptually challenging illumination conditions and achieve colour constancy (Mitterer and de Ruiter, 2008; Witzel and Gegenfurtner, 2018). Thus, the visual perception of object colours can be thought of as leveraging top-down knowledge for bottom-up pro- cessing of sensory input, in accordance with tra- ditional approaches in psychology (e.g. Colman, 2009). The integration of visual information and world knowledge in perception, however, is far from obvious, with views ranging from processing through bidirectionally connected bottom-up and top-down components to the assumption that vi- sual and conceptual representations themselves are inseparably intertwined (Kubat et al., 2009). Figure 1: Example object from VisualGenome with annotated colour attribute. The tree is described as “green”, despite of challenging illumination conditions. A lot of recent work in Language & Vision (L&V) has looked at grounding language in real- istic sensory information, e.g. images of complex, real-world scenes and objects (Bernardi et al., 2016; Kafle and Kanan, 2017). In L&V, however, the use of top-down knowledge has mostly been discussed in the context of zero-shot or few-shot learning scenarios where few or no visual instances of a par- ticular object category are available (Frome et al., 2013; Xian et al., 2018). 1 We present a simple experiment on language grounding that highlights the great potential of top-down processing even for very common words with a lot of visual instances: we learn to ground colour terms in visual representations of real-world objects and show that model predictions improve strongly when incorporating prior knowledge and assumptions about the object itself. We investi- gate visual grounding of colour terms by combin- ing bottom-up and top-down modeling components based on early and late fusion strategies, reflecting different interpretations about …

Content truncated to 500 words