Hierarchical Context-aware Network for Dense Video Event Captioning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2004–2013 August 1–6, 2021. ©2021 Association for Computational Linguistics 2004 Hierarchical Context-aware Network for Dense Video Event Captioning Lei Ji1,2,3 ∗, Xianglin Guo4∗, Haoyang Huang3,Xilin Chen1,2 1Institute of Computing Technology, CAS, Beijing, China 2University of Chinese Academy of Sciences, Beijing, China 3Microsoft Research Asia, Beijing, China 4New York University, New York, USA leiji@microsoft.com,xg893@nyu.edu haoyang.huang@microsoft.com,xlchen@ict.ac.cn Abstract Dense video event captioning aims to gener- ate a sequence of descriptive captions for each event in a long untrimmed video. Video-level context provides important information and fa- cilities the model to generate consistent and less redundant captions between events. In this paper, we introduce a novel Hierarchical Context-aware Network for dense video event captioning (HCN) to capture context from var- ious aspects. In detail, the model leverages local and global context with different mech- anisms to jointly learn to generate coherent captions. The local context module performs full interaction between neighbor frames and the global context module selectively attends to previous or future events. According to our extensive experiment on both Youcook2 and Activitynet Captioning datasets, the video- level HCN model outperforms the event-level context-agnostic model by a large margin. The code is available at https://github.com/ KirkGuo/HCN. 1 Introduction With the increase of video data uploaded online ev- ery day, the acquisition of knowledge from videos especially for Howto tasks is indispensable for peo- ple’s daily life and work. However, watching a whole long video is time-consuming. Existing tech- nologies focus on two main research directions to compact video information: video summariza- tion to trim long videos to short ones and (dense) video captioning to generate a textual description of the key events in the video. Typically for long untrimmed videos, dense video event captioning generates fine-grained captions for all events to facilitate users quickly skimming the video con- tent and enables various applications e.g. video chaptering and search inside a video. ∗Equal contribution Figure 1: A showcase of dense video event captioning. Given a video and the speech text, the task is to gener- ate event proposals and captions. Dense video event captioning (Krishna et al., 2017) and multi-modal video event captioning (Iashin and Rahtu, 2020b) aims to generate a se- quence of captions for all events regarding to uni-modality (video) or multi-modality (video + speech) inputs. Figure 1 presents a showcase, which demonstrates the challenges of this task from both vision and speech text perspective. For vision understanding, the fine-grained objects are hard to recognize due to ambiguity, occlusion, or state change. In this case, the object ”dough” is oc- cluded in event 1 and is hard to recognize from the video. However, it can be recognized from the previous neighbor video frame with a clear ap- pearance. From speech text perspective, although the speech text offers semantic concepts (Shi et al., 2019; Iashin and Rahtu, 2020b), it brings another challenge of co-reference and ellipsis in speech text due …

Content truncated to 500 words