VQD: Visual Query Detection In Natural Scenes

Proceedings of NAACL-HLT 2019, pages 1955–1961 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1955 VQD: Visual Query Detection in Natural Scenes Manoj Acharya1 Karan Jariwala1 Christopher Kanan1,2,3 1Rochester Institute of Technology 2PAIGE 3Cornell Tech {ma7583, kkj1811, kanan}@rit.edu Abstract We propose Visual Query Detection (VQD), a new visual grounding task. In VQD, a system is guided by natural language to localize a vari- able number of objects in an image. VQD is related to visual referring expression recogni- tion, where the task is to localize only one ob- ject. We describe the first dataset for VQD and we propose baseline algorithms that demon- strate the difficulty of the task compared to re- ferring expression recognition. 1 Introduction In computer vision, object detection is the task of identifying all objects from a specific closed-set of pre-defined classes by putting a bounding box around each object present in an image, e.g., in the widely used COCO dataset there are 80 object cat- egories and an algorithm must put a box around all instances of each object present in an image (Lin et al., 2014). Recent deep learning based models have significantly advanced the state-of-the-art in object detection (Ren et al., 2015b); however, many applications demand more nuanced detection of ob- jects with specific attributes or objects in relation to each other. Here, we study goal-directed object de- tection, where the set of possible valid objects is far greater than in the typical object detection problem. Specifically, we introduce the Visual Query Detec- tion (VQD) task (see Fig. 1). In VQD, a system is given a query in natural language and an image and it must produce 0–N boxes that satisfy that query. VQD has numerous applications, ranging from image retrieval to robotics. VQD is related to the visual referring expres- sion recognition (RER) task (Kazemzadeh et al., 2014); however, in RER every image has only a single correct box. In contrast, in VQD there could be no valid outputs for a query or multiple valid outputs, making the task both harder and more use- ful. As discussed later, existing RER datasets have Figure 1: Unlike VQD, object detection cannot deal with attributes and relations. In VQA, often algorithms produce the right answers due to dataset bias without ‘looking’ at relevant image regions. RER datasets have short and often ambiguous prompts, and by having only a single box as an output, they make it easier to exploit dataset biases. VQD requires goal-directed object de- tection and outputting a variable number of boxes that answer a query. multiple annotation problems and have significant language bias problems. VQD is also related to Visual Question Answering (VQA), where the task is to answer questions about images in natural lan- guage (Malinowski and Fritz, 2014; Antol et al., 2015). The key difference is that in VQD the al- gorithm must generate image bounding boxes that satisfy the query, making it less prone to the forms of bias that plague VQA datasets. We …

Content truncated to 500 words