Stating the Obvious: Extracting Visual Common Sense Knowledge

Proceedings of NAACL-HLT 2016, pages 193–198, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Stating the Obvious: Extracting Visual Common Sense Knowledge Mark Yatskar1, Vicente Ordonez2, Ali Farhadi1,2 1Computer Science & Engineering, University of Washington, Seattle, WA 2Allen Institute for Artificial Intelligence (AI2), Seattle, WA [my89, ali]@cs.washington.edu, vicenteor@allenai.org Abstract Obtaining common sense knowledge using current information extraction techniques is extremely challenging. In this work, we in- stead propose to derive simple common sense statements from fully annotated object detec- tion corpora such as the Microsoft Common Objects in Context dataset. We show that many thousands of common sense facts can be extracted from such corpora at high quality. Furthermore, using WordNet and a novel sub- modular k-coverage formulation, we are able to generalize our initial set of common sense assertions to unseen objects and uncover over 400k potentially useful facts. 1 Introduction How can we discover that bowls can hold broc- coli, that if a knife touches a cake then a person is probably cutting cake, or that cutlery can be on dining tables? We propose to leverage the effort of computer vision researchers in creating large scale datasets for object detection and use these resources instead to extract symbolic representations of visual common sense. The knowledge we compile is phys- ical, not commonly covered in text and more exhaus- tive than what people can usually produce. Our focus is particularly on visual common sense, defined as the information about spatial and func- tional properties of entities in the world. We pro- pose to extract three types of knowledge from the Microsoft Common Objects in Context dataset (Lin et al., 2014) (MS-COCO), consisting of 300,000 images, covering 80 objects, with object segments and natural language captions. First, we find spa- tial relations, e.g. holds(bed, dog), from outlines of co-occurring objects. Next, we construct entail- ment rules like holds(bed, dog) ⇒ laying-on(dog, bed) by associating spatial relations with text in captions. Finally, we uncover general facts such as holds(furniture, domestic animal), applicable to object types not present in MS-COCO by using WordNet (Miller, 1995) and a novel submodular k- coverage formulation. Evaluations using crowdsourcing show our meth- ods can discover many thousands of high qual- ity explicit statements of visual common sense. While some of this knowledge can be potentially ex- tracted from text (Vanderwende, 2005), we found that from our top 100 extracted spatial relations, e.g. holds(bed, dog), only 4 are present in some form in the AtLocation relations in the popular Con- ceptNet (Speer and Havasi, 2013) knowledge base. This shows that the knowledge we derive provides complimentary information for other more general knowledge bases. Such common sense facts have proved useful for query expansion (Kotov and Zhai, 2012; Bouchoucha et al., 2013) and could benefit en- tailment (Dagan et al., 2010), grounded entailment (Bowman et al., 2015), or visual recognition tasks (Zhu et al., 2014). 2 Related Work Common sense knowledge has been predominately created directly from human input or extracted from text (Lenat et …

Content truncated to 500 words