Zero-Shot Activity Recognition with Verb Attribute Induction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 946–958 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Zero-Shot Activity Recognition with Verb Attribute Induction Rowan Zellers and Yejin Choi Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA {rowanz,yejin}@cs.washington.edu Abstract In this paper, we investigate large-scale zero-shot activity recognition by modeling the visual and linguistic attributes of ac- tion verbs. For example, the verb “salute” has several properties, such as being a light movement, a social act, and short in dura- tion. We use these attributes as the internal mapping between visual and textual rep- resentations to reason about a previously unseen action. In contrast to much prior work that assumes access to gold standard attributes for zero-shot classes and focuses primarily on object attributes, our model uniquely learns to infer action attributes from dictionary definitions and distributed word representations. Experimental re- sults confirm that action attributes inferred from language can provide a predictive signal for zero-shot prediction of previ- ously unseen activities. 1 Introduction We study the problem of inferring action verb at- tributes based on how the word is defined and used in context. For example, given a verb such as “swig” shown in Figure 1, we want to infer var- ious properties of actions such as motion dynam- ics (moderate movement), social dynamics (soli- tary act), body parts involved (face, arms, hands), and duration (less than 1 minute) that are generally true for the range of actions that can be denoted by the verb “swig”. Our ultimate goal is to improve zero-shot learn- ing of activities in computer vision: predicting a previously unseen activity by integrating back- ground knowledge about the conceptual properties of actions. For example, a computer vision system may have seen images of “drink” activities during Dictionary definitions drool drink chug sip swig To drink liquid in great gulps to drink a large amount especially of beer To drink in small quantities To let run from the mouth To take into the mouth and swallow a liquid medium motion more solitary N/A activity more solitary more solitary medium motion achieve ment solitary or social activity achieve ment activity solitary or social no motion low motion low motion Word embeddings zero-shot image … sip: 10% chug: 25% swig: 65% Distribution over Zero-shot labels attribute-embedding space Verb attribute induction from language A) B) Zero-shot activity recognition training images aspect motion uses head has effect on object social uncork drool lick drink Figure 1: An overview of our task. Our goal is twofold. A: we seek to use use distributed word embeddings in tandem with dictionary definitions to obtain a high level understanding of verbs. B: we seek to use these predicted attributes to allow a classifier to recognize a broader set of activities than what was seen in training time. training, but not “swig”. Ideally, the system should infer the likely visual characteristics of “swig” us- ing world knowledge implicitly available in …

Content truncated to 500 words