Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat

Proceedings of NAACL-HLT 2019, pages 2578–2587 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2578 Beyond Task Success: A Closer Look at Jointly Learning to See, Ask, and GuessWhat Ravi Shekhar†, Aashish Venkatesh∗, Tim Baumg¨artner∗, Elia Bruni∗, Barbara Plank♥, Raffaella Bernardi† and Raquel Fern´andez∗ †University of Trento, ∗University of Amsterdam,♥IT University of Copenhagen ravi.shekhar@unitn.it aashishv@outlook.com baumgaertner.t@gmail.com elia.bruni@gmail.com bapl@itu.dk raffaella.bernardi@unitn.it raquel.fernandez@uva.nl Abstract We propose a grounded dialogue state encoder which addresses a foundational issue on how to integrate visual grounding with dialogue system components. As a test-bed, we fo- cus on the GuessWhat?! game, a two-player game where the goal is to identify an object in a complex visual scene by asking a sequence of yes/no questions. Our visually-grounded encoder leverages synergies between guessing and asking questions, as it is trained jointly using multi-task learning. We further enrich our model via a cooperative learning regime. We show that the introduction of both the joint architecture and cooperative learning lead to accuracy improvements over the baseline sys- tem. We compare our approach to an alter- native system which extends the baseline with reinforcement learning. Our in-depth analysis shows that the linguistic skills of the two mod- els differ dramatically, despite approaching comparable performance levels. This points at the importance of analyzing the linguistic output of competing systems beyond numeric comparison solely based on task success.1 1 Introduction Over the last few decades, substantial progress has been made in developing dialogue systems that ad- dress the abilities that need to be put to work dur- ing conversations: Understanding and generating natural language, planning actions, and tracking the information exchanged by the dialogue partic- ipants. The latter is particularly critical since, for communication to be effective, participants need to represent the state of the dialogue and the com- mon ground established through the conversation (Stalnaker, 1978; Lewis, 1979; Clark, 1996). In addition to the challenges above, dialogue is often situated in a perceptual environment. In 1Equal contribution by R. Shekhar and A. Venkatesh. Figure 1: Our questioner model with a single visually grounded dialogue state encoder. this study, we develop a dialogue agent that builds a representation of the context and the dialogue state by integrating information from both the vi- sual and linguistic modalities. We take the Guess- What?! game (de Vries et al., 2017) as our test- bed, a two-player game where a Questioner faces the task of identifying a target object in a visual scene by asking a series of yes/no questions to an Oracle. We model the agent in the Questioner’s role. To model the Questioner, previous work relies on two independent models to learn to ask ques- tions and to guess the target object, each equipped with its own encoder (de Vries et al., 2017; Strub et al., 2017; Zhu et al., 2017; Lee et al., 2017; Shekhar et al., 2018; Zhang et al., 2018). We propose an end-to-end architecture with a single visually-grounded dialogue state encoder (cf. Fig- ure 1). Our system is …

Content truncated to 500 words