A vision-grounded dataset for predicting typical locations for verbs

A vision-grounded dataset for predicting typical locations for verbs Nelson Mukuze1, Anna Rohrbach3,4, Vera Demberg2, Bernt Schiele3 1Visual Meta GmbH 2Saarland University 3Max Planck Institute for Informatics, Saarland Informatics Campus 4EECS, UC Berkeley nelson.mukuze@visual-meta.com, vera@coli.uni-saarland.de, {arohrbach, schiele}@mpi-inf.mpg.de Abstract Information about the location of an action is often implicit in text, as humans can infer it based on common sense knowledge. Today’s NLP systems however struggle with inferring information that goes beyond what is explicit in text. Selectional preference estimation based on large amounts of data provides a way to infer prototypical role fillers, but text-based systems tend to underestimate the probability of the most typical role fillers. We here present a new dataset containing thematic fit judgments for 2,000 verb/location pairs. This dataset can be used for evaluating text-based, vision-based or multimodal inference systems for the typicality of an event’s location. We additionally provide three thematic fit baselines for this dataset: a state-of-the-art neural networks based thematic fit model learned from linguistic data, a model estimating typical locations based on the MSCOCO dataset and a simple combination of the systems. Keywords: human judgments, thematic fit, vision 1. Introduction Most of automatic language understanding today is based on information that is explicitly stated in text. How- ever, tasks that require the ability to make additional common sense inferences are increasingly coming into focus (Levesque et al., 2011; Roemmele et al., 2011; Mostafazadeh et al., 2016). We here propose a dataset which addresses a sub-task towards this goal, i.e. inferring typical locations for a given verb. This task is related to selectional preference tasks, and taps into the kind of infer- ences humans make when comprehending language. The task consists of predicting a location, given other informa- tion from the text (here: the predicate), see also Baroni and Lenci (2010; Sayeed et al. (2015; Tilk et al. (2016). Given the verb to eat, a good thematic role filling model would for instance prefer the location restaurant over office. Previous work has found that locations are a particularly difficult role to predict (Sayeed et al., 2016). This could be due to the fact that location modifiers are often omitted in text when the lo- cation is inferable from common sense knowledge, known as reporting bias (Gordon and Van Durme, 2013; Misra et al., 2016). It is hence a logical next step to use multimodal data, and in particular, data from vision. However, there are to date no datasets which can be used to evaluate the con- tribution of a vision model, as the only existing location fit dataset is very small (Ferretti et al., 2001), comprising only 277 verb/location pairs, and includes many rare verbs. We here present a new dataset based on 20,000 human judg- ments for a total of 2,000 verb/location pairs. The dataset was specifically constructed to evaluate the contribution of both visual and linguistic information for learning common sense knowledge about locations. It builds on MS COCO (Lin et al., 2014), a dataset of images with captions; lo- cations …

Content truncated to 500 words