Grounding Action Descriptions in Videos

Transactions of the Association for Computational Linguistics, 1 (2013) 25–36. Action Editor: Hal Daum´e III. Submitted 10/2012; Published 3/2013. c⃝2013 Association for Computational Linguistics. Grounding Action Descriptions in Videos Michaela Regneri ∗, Marcus Rohrbach ⋄, Dominikus Wetzel ∗, Stefan Thater ∗, Bernt Schiele ⋄ and Manfred Pinkal ∗ ∗ Department of Computational Linguistics, Saarland University, Saarbr¨ucken, Germany (regneri|dwetzel|stth|pinkal)@coli.uni-saarland.de ⋄ Max Planck Institute for Informatics, Saarbr¨ucken, Germany (rohrbach|schiele)@mpi-inf.mpg.de Abstract Recent work has shown that the integration of visual information into text-based models can substantially improve model predictions, but so far only visual information extracted from static images has been used. In this paper, we consider the problem of grounding sentences describing actions in visual information ex- tracted from videos. We present a general purpose corpus that aligns high quality videos with multiple natural language descriptions of the actions portrayed in the videos, together with an annotation of how similar the action descriptions are to each other. Experimental results demonstrate that a text-based model of similarity between actions improves substan- tially when combined with visual information from videos depicting the described actions. 1 Introduction The estimation of semantic similarity between words and phrases is a basic task in computational semantics. Vector-space models of meaning are one standard approach. Following the distributional hy- pothesis, frequencies of context words are recorded in vectors, and semantic similarity is computed as a proximity measure in the underlying vector space. Such distributional models are attractive because they are conceptually simple, easy to implement and relevant for various NLP tasks (Turney and Pan- tel, 2010). At the same time, they provide a sub- stantially incomplete picture of word meaning, since they ignore the relation between language and extra- linguistic information, which is constitutive for lin- guistic meaning. In the last few years, a growing amount of work has been devoted to the task of grounding meaning in visual information, in par- ticular by extending the distributional approach to jointly cover texts and images (Feng and Lapata, 2010; Bruni et al., 2011). As a clear result, visual information improves the quality of distributional models. Bruni et al. (2011) show that visual infor- mation drawn from images is particularly relevant for concrete common nouns and adjectives. A natural next step is to integrate visual infor- mation from videos into a semantic model of event and action verbs. Psychological studies have shown the connection between action semantics and videos (Glenberg, 2002; Howell et al., 2005), but to our knowledge, we are the first to provide a suitable data source and to implement such a model. The contribution of this paper is three-fold: • We present a multimodal corpus containing textual descriptions aligned with high-quality videos. Starting from the video corpus of Rohrbach et al. (2012b), which contains high- resolution video recordings of basic cooking tasks, we collected multiple textual descrip- tions of each video via Mechanical Turk. We also provide an accurate sentence-level align- ment of the descriptions with their respective videos. We expect the corpus to be a valu- able …

Content truncated to 500 words