Object Naming in Language and Vision: A Survey and a New Dataset

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5792–5801 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5792 Object Naming in Language and Vision: A Survey and a New Dataset Carina Silberer, Sina Zarrieß, Gemma Boleda Universitat Pompeu Fabra, University of Jena, Universitat Pompeu Fabra Barcelona (Spain), Jena (Germany), Barcelona (Spain) sina.zarriess@uni-jena.de {carina.silberer, gemma.boleda}@upf.edu Abstract People choose particular names for objects, such as dog or puppy for a given dog. Object naming has been studied in Psycholinguistics, but has received relatively little attention in Computational Linguistics. We review resources from Language and Vision that could be used to study object naming on a large scale, discuss their shortcomings, and create a new dataset that affords more opportunities for analysis and modeling. Our dataset, ManyNames, provides 36 name annotations for each of 25K objects in images selected from VisualGenome. We highlight the challenges involved and provide a preliminary analysis of the ManyNames data showing that there is a high level of agreement in naming, on average. At the same time, the average number of name types associated with an object is much higher in our dataset than in existing corpora for Language and Vision, such that ManyNames provides a rich resource for studying phenomena like hierarchical variation (chihuahua vs. dog), which has been discussed at length in the theoretical literature, and other less well studied phenomena like cross-classification (cake vs. dessert). Keywords: object naming, language and vision, computer vision 1. Introduction A central issue in Language & Vision (L&V) is how speak- ers refer to objects. This is most prominent for referring expression generation and interpretation (Kazemzadeh et al., 2014; Mao et al., 2015; Yu et al., 2016), but it also pervades virtually any other L&V task, such as caption generation or visual dialogue (Fang et al., 2015; Devlin et al., 2015; Das et al., 2017; De Vries et al., 2017). One of the central compo- nents of referring expressions are object names; for instance, speakers may name the left object in Figure 1 cake, food, or dessert, a.o. This aspect of reference has been understudied in Computational Linguistics and L&V; as a consequence, it is not clear to what extent current L&V models capture human naming behavior. In the same way that it has proven useful to model referring expressions in visual scenes in an isolated fashion, for which systems are required to integrate and reason over the visual scene and context objects in which an object is presented, we believe there is value in modeling object names on their own. Specifically, questions that need addressing regard- ing object naming are (1) how much naming variation is attested and what factors drive the choice of a name (object category? individual properties of the object? context?); see e.g. Rohde et al. (2012) and Graf et al. (2016); (2) how to make L&V models more human-like with respect to naming, improving the design of L&V architectures (e.g., Lazaridou et al. (2015); Ordonez …

Content truncated to 500 words