CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7625–7641 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7625 CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning Alessandro Suglia1, Ioannis Konstas1, Andrea Vanzo1, Emanuele Bastianelli1, Desmond Elliott2, Stella Frank3, and Oliver Lemon1 1Heriot-Watt University, Edinburgh, UK 2University of Copenhagen, Copenhagen, Denmark 3University of Edinburgh, Edinburgh, UK 1{as247,i.konstas,a.vanzo,e.bastianelli,o.lemon}@hw.ac.uk 2de@di.ku.dk 3stella.frank@ed.ac.uk Abstract Approaches to Grounded Language Learning typically focus on a single task-based final per- formance measure that may not depend on de- sirable properties of the learned hidden rep- resentations, such as their ability to predict salient attributes or to generalise to unseen sit- uations. To remedy this, we present GROLLA, an evaluation framework for Grounded Lan- guage Learning with Attributes with three sub- tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero- shot evaluation. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations, in particular concerning attribute grounding. To this end, we extend the original GuessWhat?! dataset by including a semantic layer on top of the perceptual one. Specifically, we enrich the Vi- sualGenome scene graphs associated with the GuessWhat?! images with abstract and situ- ated attributes. By using diagnostic classifiers, we show that current models learn representa- tions that are not expressive enough to encode object attributes (average F1 of 44.27). In addi- tion, they do not learn strategies nor represen- tations that are robust enough to perform well when novel scenes or objects are involved in gameplay (zero-shot best accuracy 50.06%). 1 Introduction Several grounded language learning tasks have been proposed to capture perceptual aspects of lan- guage (Shekhar et al., 2017; Hudson and Manning, 2019; Suhr et al., 2019; Agrawal et al., 2018). How- ever, the advances in this field have been primarily driven by the final performance measures and less on the grounding capability of the models. In fact, in some cases, high-performance models exploit dataset biases to achieve high scores on the final task (Zhang et al., 2016; Agrawal et al., 2016). In Turn Question Answer 1 is it an appliance? yes 2 does it blend? no 3 is it the oven? no 4 is it the microwave? yes left-of left-of microwave oven food mixer is appliance has buttons is white is appliance has buttons is white is appliance has buttons usedto_blend is white Legend abstract situated Figure 1: Every gameplay in the CompGuess- What?! benchmark has a reference scene that is mapped to a scene graph composed of objects represented in terms of abstract and situated attributes. the literature, several methods have been proposed to analyse what kind of information is captured by neural network representations (K´ad´ar et al., 2017; Belinkov and Glass, 2019). Most of these works examine the hidden state representations learned by models trained on only textual data. However, many aspects of human semantic representations are grounded in perceptual experience (Andrews et al., 2009; Riordan and Jones, 2011). This …

Content truncated to 500 words