Exploiting Language Models for Visual Recognition

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 769–779, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Exploiting language models for visual recognition Dieu-Thu Le DISI, University of Trento Povo, 38123, Italy dle@disi.unitn.it Jasper Uijlings DISI, University of Trento Povo, 38123, Italy jrr@disi.unitn.it Raffaella Bernardi DISI, University of Trento Povo, 38123, Italy bernardi@disi.unitn.it Abstract The problem of learning language models from large text corpora has been widely stud- ied within the computational linguistic com- munity. However, little is known about the performance of these language models when applied to the computer vision domain. In this work, we compare representative models: a window-based model, a topic model, a distri- butional memory and a commonsense knowl- edge database, ConceptNet, in two visual recognition scenarios: human action recog- nition and object prediction. We examine whether the knowledge extracted from texts through these models are compatible to the knowledge represented in images. We de- termine the usefulness of different language models in aiding the two visual recognition tasks. The study shows that the language models built from general text corpora can be used instead of expensive annotated images and even outperform the image model when testing on a big general dataset. 1 Introduction Computational linguistics have created many tools for automatic knowledge acquisition which have been successfully applied in many tasks inside the language domain, such as question answering, ma- chine translation, semantic web, etc. In this paper we ask whether such knowledge generalizes to the observed reality outside the language domain, where we use well-known image datasets as a proxy for ob- served reality. In particular, we aim to determine which language model yields knowledge that is most suitable for use in Computer Vision. Therefore we test a variety of language models and a linguistically mined knowl- edge base within two computer vision scenarios: Human action recognition : Recognizing <subject, verb, object> triples based on objects (e.g., car, horse) and scenes (the place that the actions occur, e.g., countryside, forest, office) recognized in images. In this scenario, we only consider images with human actions so the “human” subject is always present. Objects in context : Predicting the most likely identity of an object given its context as ex- pressed in terms of co-occurring objects. Computer vision can greatly benefit from natural language processing as learning from images re- quires a prohibitively expensive annotation effort. A major goal of natural language processing is to ob- tain general knowledge from text and in this paper we test which model provides the best knowledge for use in the visual domain. Within the two visual scenarios, we compare three state-of-the-art language models and a knowledge base: (1) A window-based model, which counts co-occurrence frequencies within a fixed window; (2) R-LDA (S´eaghdha, 2010), an extension of LDA that enables generation of joint probabilities; (3) TypeDM (Baroni and Lenci, 2010), a strong Distri- butional Memory model; (4) ConceptNet (Speer and Havasi, 2013), an automatically generated semantic graph containing concepts with their …

Content truncated to 500 words