Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games

Proceedings of the 28th International Conference on Computational Linguistics, pages 1090–1102 Barcelona, Spain (Online), December 8-13, 2020 1090 Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games Alessandro Suglia1, Antonio Vergari2, Ioannis Konstas1, Yonatan Bisk3, Emanuele Bastianelli1, Andrea Vanzo1, and Oliver Lemon1 1Heriot-Watt University, Edinburgh, UK 2University of California, Los Angeles, USA 3Carnegie Mellon University, Pittsburgh, USA 1{as247,i.konstas,a.vanzo,e.bastianelli,o.lemon}@hw.ac.uk 2aver@cs.ucla.edu, 3ybisk@cs.cmu.edu Abstract In visual guessing games, a Guesser has to identify a target object in a scene by asking ques- tions to an Oracle. An effective strategy for the players is to learn conceptual representations of objects that are both discriminative and expressive enough to ask questions and guess cor- rectly. However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at train- ing and inference time. This provides an unnatural performance advantage when categories at inference time match those at training time, and it causes models to fail in more realistic “zero- shot” scenarios where out-of-domain object categories are involved. To overcome this issue, we introduce a novel “imagination” module based on Regularized Auto-Encoders, that learns context-aware and category-aware latent embeddings without relying on category labels at infer- ence time. Our imagination module outperforms state-of-the-art competitors by 8.26% gameplay accuracy in the CompGuessWhat?! zero-shot scenario (Suglia et al., 2020), and it improves the Oracle and Guesser accuracy by 2.08% and 12.86% in the GuessWhat?! benchmark, when no gold categories are available at inference time. The imagination module also boosts reasoning about object properties and attributes. 1 Introduction Humans do not learn conceptual representations from language alone, but from a wide range of situa- tional information (Beinborn et al., 2018; Bisk et al., 2020) as highlighted also by property-listing exper- iments (McRae et al., 2005). When humans experience the concept of “boat”, they simulate a new rep- resentation by reactivating and aggregating multi-modal representations that reside in their memory and are associated with the concept of “boat” (e.g., what a boat looks like, the action of sailing, etc) (Barsa- lou, 2008). This simulation process is called perceptual simulation. Therefore, it is no wonder that recent trends in learning conceptual representations adopt multi-modal and holistic approaches (Bruni et al., 2014) wherein abstract distributional lexical representations (Landauer and Dumais, 1997; Laurence and Margolis, 1999) learned from text corpora are augmented or refined with perceptual information for concrete and context-aware representations built from visual (Kiela et al., 2018; Lazaridou et al., 2015), olfactory (Kiela et al., 2015), or auditory (Kiela and Clark, 2015) modalities. Language games between AI agents, inspired by Wittgenstein’s Language Games among hu- mans (Wittgenstein et al., 1953), are an excellent test bed for such approaches since concepts are ex- pected to emerge when agents are required to communicate to solve specific tasks in specific environ- ments. GuessWhat?! (De Vries et al., 2017) is a prototypical language game of this kind: a Guesser has to identify a target object in a …

Content truncated to 500 words