Extracting Latent Attributes from Video Scenes Using Text as Background Knowledge

Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), pages 121–131, Dublin, Ireland, August 23-24 2014. Extracting Latent Attributes from Video Scenes Using Text as Background Knowledge Anh Tran, Mihai Surdeanu, Paul Cohen University of Arizona {trananh, msurdeanu, prcohen}@email.arizona.edu Abstract We explore the novel task of identify- ing latent attributes in video scenes, such as the mental states of actors, using only large text collections as background knowledge and minimal information about the videos, such as activity and actor types. We formalize the task and a measure of merit that accounts for the semantic re- latedness of mental state terms. We de- velop and test several largely unsupervised information extraction models that iden- tify the mental states of human partici- pants in video scenes. We show that these models produce complementary informa- tion and their combination significantly outperforms the individual models as well as other baseline methods. 1 Introduction “Labeling a narrowly avoided vehicular manslaughter as approach(car, person) missing something.”1 The recognition of ac-is tivities, participants, and objects in videos has advanced considerably in recent years (Li et al., 2010; Poppe, 2010; Weinland et al., 2011; Yang and Ramanan, 2011; Ng et al., 2012). However, identifying latent attributes of scenes, such as the mental states of human participants, has not been addressed. Latent attributes matter: If a video surveillance system detects one person chasing another, the response from law enforcement should be radically different if the people are happy (e.g., children playing) or afraid and angry (e.g., a person running from an assailant). This work is licenced under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organizers. License details: http: //creativecommons.org/licenses/by/4.0/ 1James Donlon, former manager of DARPA’s Mind’s Eye program, personal communication. Attributes that are latent in visual representa- tions are often explicit in textual representations. This suggests a novel method for inferring latent attributes: Use explicit features of videos to query text corpora, and from the resulting texts extract attributes that are latent in the videos, such as men- tal states. The contributions of this work are: 1: We formalize the novel task of latent attribute identification from video scenes, focusing on the identification of actors’ mental states. The input for the task is contextual information about the scene, such as detections about the activity (e.g., chase) and actor types (e.g., policeman or child), and the output is a distribution over mental state labels. We show that gold standard annotations for this task can be reliably generated using crowd sourcing. We define a novel evaluation measure, called constrained weighted similarity-aligned F1 score, that accounts for both the differences be- tween mental state distributions and the seman- tic relatedness of mental state terms (e.g., partial credit is given for irate when the target is angry). 2: We propose several robust and largely unsuper- vised information extraction (IE) models for iden- tifying the mental state labels of human partici- pants in a scene, given solely the activity and actor …

Content truncated to 500 words