Humans Meet Models on Object Naming: A New Dataset and Analysis

Proceedings of the 28th International Conference on Computational Linguistics, pages 1893–1905 Barcelona, Spain (Online), December 8-13, 2020 1893 Humans Meet Models on Object Naming: A New Dataset and Analysis Carina Silberer†‡ Sina Zarrieß⋄ Matthijs Westera‡ Gemma Boleda‡ †University of Stuttgart ⋄Friedrich Schiller University Jena ‡Universitat Pompeu Fabra CarinaSilberer@gmail.com sina.zarriess@uni-jena.de matthijs.westera@gmail.com gemma.boleda@upf.edu Abstract We release ManyNames v2 (MN v2), a verified version of an object naming dataset that contains dozens of valid names per object for 25K images. We analyze issues in the data collection method originally employed, standard in Language & Vision (L&V), and find that the main source of noise in the data comes from simulating a naming context solely from an image with a target object marked with a bounding box, which causes subjects to sometimes disagree regarding which object is the target. We also find that both the degree of this uncertainty in the original data and the amount of true naming variation in MN v2 differs substantially across object domains. We use MN v2 to analyze a popular L&V model and demonstrate its effectiveness on the task of object naming. However, our fine-grained analysis reveals that what appears to be human-like model behavior is not stable across domains, e.g., the model confuses people and clothing objects much more frequently than humans do. We also find that standard evaluations underestimate the actual effectiveness of the naming model: on the single-label names of the original dataset (Visual Genome), it obtains −27% accuracy points than on MN v2, that includes all valid object names. 1 Introduction Research on object naming (Ordonez et al., 2016; Graf et al., 2016; Eisape et al., 2020), such as the linguistic analysis of the naming behavior of humans and computational models, requires natural and reliable object naming data. Such data should contain naturally occurring naming variation, i.e., that the same object can very often be called by different names. For instance, a duck can be called duck, bird, animal, etc. At the same time, in cases where a naming dataset provides multiple names for a given object, it is important to verify that this apparent naming variation is not a mere consequence of, for instance, lexical errors or different people naming different objects in the same scene. In this work, we assess the factors that affect the collection of object naming data in the typical Language & Vision (L&V) setup, and test whether these factors impact the accuracy or apparent accuracy of a L&V object labeling model. Existing work in L&V has relied on data collection methods that prompt natural language users to freely talk about or refer to particular objects in an image (Kazemzadeh et al., 2014; Yu et al., 2016; Silberer et al., 2020). Common to these methods is the use of images as a proxy for the actual context of language use (i.e., the real world), and the use of bounding boxes drawn onto the image to indicate the target object that is to be named. These two common aspects, essentially …

Content truncated to 500 words