Identifying Visible Actions in Lifestyle Vlogs

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6406–6417 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6406 Identifying Visible Actions in Lifestyle Vlogs Oana Ignat1, Laura Burdick1, Jia Deng2, Rada Mihalcea1 1University of Michigan, 2Princeton University {oignat,wenlaura,mihalcea}@umich.edu, jiadeng@cs.princeton.edu Abstract We consider the task of identifying human ac- tions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech de- scription of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages informa- tion derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time. 1 Introduction There has been a surge of recent interest in detect- ing human actions in videos. Work in this space has mainly focused on learning actions from artic- ulated human pose (Du et al., 2015; Vemulapalli et al., 2014; Zhang et al., 2017) or mining spatial and temporal information from videos (Simonyan and Zisserman, 2014; Wang et al., 2016). A num- ber of resources have been produced, including Action Bank (Sadanand and Corso, 2012), NTU RGB+D (Shahroudy et al., 2016), SBU Kinect In- teraction (Yun et al., 2012), and PKU-MMD (Liu et al., 2017). Most research on video action detection has gathered video information for a set of pre-defined actions (Fabian Caba Heilbron and Niebles, 2015; Real et al., 2017; Kay et al., 2017), an approach known as explicit data gathering (Fouhey et al., 2018). For instance, given an action such as “open door,” a system would identify videos that include a visual depiction of this action. While this ap- proach is able to detect a specific set of actions, whose choice may be guided by downstream ap- plications, it achieves high precision at the cost of low recall. In many cases, the set of predefined ac- tions is small (e.g., 203 activity classes in Fabian Caba Heilbron and Niebles 2015), and for some actions, the number of visual depictions is very small. An alternative approach is to start with a set of videos, and identify all the actions present in these videos (Damen et al., 2018; Bregler, 1997). This approach has been referred to as implicit data gathering, and it typically leads to the identifica- tion of a larger number of actions, possibly with a small number of examples per action. In this paper, we use an implicit data gather- ing approach to label human activities in videos. To the best of our knowledge, we are the first to explore video action recognition using both tran- scribed audio and video information. We focus on the popular genre of lifestyle vlogs, which con- sist of videos of people demonstrating routine ac- tions while …

Content truncated to 500 words