An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2135–2144 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2135 An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games Alessandro Suglia1, Yonatan Bisk2, Ioannis Konstas1, Antonio Vergari3, Emanuele Bastianelli1, Andrea Vanzo1, and Oliver Lemon1 1Heriot-Watt University, Edinburgh, UK 2Carnegie Mellon University, Pittsburgh, USA 3University of California, Los Angeles, USA 1{as247,i.konstas,e.bastianelli,a.vanzo,o.lemon}@hw.ac.uk 2ybisk@cs.cmu.edu, 3aver@cs.ucla.edu Abstract Guessing games are a prototypical instance of the “learning by interacting” paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP down- stream tasks such as Visual Question An- swering (VQA). We propose two ways to ex- ploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learn- ing (SPIEL). We evaluate the ability of both procedures to generalise: an in-domain eval- uation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL. 1 Background & Related Work Learning a language requires interacting with both the environment and other agents (Bisk et al., 2020). Language games represent one common example of this (Wittgenstein et al., 1953), as seen by the important role of play in L1 child language acqui- sition (Hainey et al., 2016) as well as L2 learn- ers (Godwin-Jones, 2014). Among the language games defined in the litera- ture (Steels, 2015), guessing games represent the first step in a curriculum for language learning. For example, in GuessWhat?! (de Vries et al., 2017), two agents interact with each other: a Questioner generates questions aimed at finding a hidden ob- ject in the scene and an Oracle, aware of the target object, answers the questions supporting the Ques- tioner in playing the game. Different from other language games (Das et al., 2017), guessing games have a specific goal which represents a clear in- centive for learning. In addition, they require that the Questioner masters both natural language gen- eration and understanding with a focus on object categories and attributes. For humans, concepts learned in this way are generic and generalisable to new tasks and domains where grounded reasoning is important (Hampton, 1979). However, how well can AI agents generalise with concepts acquired from visual guessing games? The literature has not explored if representations built from self-play are transferable, focusing in- stead on large scale self-supervised learning. For instance, large scale image captioning datasets have been used to train multi-modal Transformers (Lu et al., 2019; Li et al., 2019; Tan and Bansal, 2019; Chen et al., 2019). Multi-task learning (Lu et al., 2020) has been used to leverage the diversity of training signals provided …

Content truncated to 500 words