Visual Goal-Step Inference using wikiHow

Visual Goal-Step Inference using wikiHow Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, Chris Callison-Burch Department of Computer and Information Science, University of Pennsylvania {yueyang1,artemisp,lyuqing,zharry,myatskar,ccb}@seas.upenn.edu Abstract Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human ac- tivities. Past work in NLP has examined the task of goal-step inference for text. We intro- duce the visual analogue. We propose the Vi- sual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset har- vested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the- art multimodal models. Moreover, the mul- timodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accu- racy by 15 - 20%. Our task will facilitate mul- timodal reasoning about procedural events. 1 Introduction Recently, there has been growing attention on the representation of complex events, with renewed interest in script learning and commonsense rea- soning (Park and Motahari Nezhad, 2018; Mujtaba and Mahapatra, 2019; Li et al., 2020). One aspect of event representation is the relationship between high-level goals and the steps involved (Zhang et al., 2020b,a). For example, given a goal (e.g. “change a tire”), an intelligent system should be able to infer what steps need to be taken to accom- plish the goal (e.g. “place the jack under the car”, “raise the jack”). In most work, events are repre- sented as text (Zellers et al., 2018; Coucke et al., 2018; Zhang et al., 2019), while they could have different modalities in the real world. Learning goal-step relations in a multimodal fashion is an interesting challenge since it requires reasoning beyond image captioning. We contend that multimodal event representation learning will have interesting implications for tasks such as Figure 1: An example Visual Goal-Step Inference Task: given a text goal (bake fish), select the image (C) that represents a step towards that goal. schema learning (Li et al., 2020, 2021) to miti- gate reporting bias (Gordon and Van Durme, 2013) since steps are often not explicitly mentioned in a body of text. For instance, if a robot is asked to “get a slice of cake,” it has to know that it should “take the cake out of the box”, “cut a slice”, “put it on a plate”, and then “take the plate to the user”. Such steps are commonsense to people and thus rarely specified explicitly, making them hard to infer from textual data. However, with multimodal learning, we could obtain such details from visual signals. This multimodal goal-step relation could also be used for vision-enabled dialog systems1 to recog- nize what task a user is completing and provide helpful recommendations. We propose a new task called Visual Goal-Step Inference (VGSI): given a textual goal and multi- ple images representing candidate events, a model must choose one …

Content truncated to 500 words