WhyAct: Identifying Action Reasons in Lifestyle Vlogs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4770–4785 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4770 WHYACT: Identifying Action Reasons in Lifestyle Vlogs Oana Ignat and Santiago Castro and Hanwen Miao and Weiji Li and Rada Mihalcea Computer Science and Engineering University of Michigan {oignat,sacastro,hwmiao,weijili,mihalcea}@umich.edu Abstract We aim to automatically identify human ac- tion reasons in online videos. We focus on the widespread genre of lifestyle vlogs, in which people perform actions while verbally describing them. We introduce and make pub- licly available the WHYACT dataset, consist- ing of 1,077 visual actions manually annotated with their reasons. We describe a multimodal model that leverages visual and textual infor- mation to automatically infer the reasons cor- responding to an action presented in the video. 1 Introduction Significant research effort has been recently de- voted to the task of action recognition (Carreira and Zisserman, 2017; Shou et al., 2017; Tran et al., 2018; Chao et al., 2018; Girdhar et al., 2019; Fe- ichtenhofer et al., 2019). Action recognition works well when applied to well defined/constrained sce- narios, such as people following scripts and instruc- tions (Sigurdsson et al., 2016; Miech et al., 2019; Tang et al., 2019), performing sports (Soomro et al., 2012; Karpathy et al., 2014) or cooking (Rohrbach et al., 2012; Damen et al., 2018, 2020; Zhou et al., 2018). At the same time however, action recogni- tion is limited and error-prone once the application space is opened to everyday life. This indicates that current action recognition systems rely mostly on pattern memorization and do not effectively un- derstand the action, which makes them fragile and unable to adapt to new settings (Sigurdsson et al., 2017; Kong and Fu, 2018). Research on how to improve action recognition in videos (Sigurdsson et al., 2017) shows that recognition systems for ac- tions with known intent have a significant increase in performance, as knowing the reason for perform- ing an action is an important step for understanding that action (Tosi, 1991; Gilovich et al., 2002). In contrast to action recognition, action causal reasoning research is just emerging in computa- tional applications (Vondrick et al., 2016; Yeo et al., 2018; Zhang et al., 2021; Fang et al., 2020). Causal reasoning has direct applications on many real- life settings, for instance to understand the conse- quences of events (e.g., if “there is clutter,” “clean- ing” is required), or to enable social reasoning (e.g., when “guests are expected,” “cleaning” may be needed – see Figure 1). Most of the work to date on causal systems has relied on the use of semantic parsers to identify reasons (He et al., 2017), how- ever this approach does not work well on more realistic every-day settings. As an example, con- sider the statement “This is a mess and my friends are coming over. I need to start cleaning.” Cur- rent causal systems are unable to identify “this is a mess” and “friends are coming over” as reasons, and are thus failing to …

Content truncated to 500 words