Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2001–2011, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2001 Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning Tao Jin, Siyu Huang∗, Yingming Li, and Zhongfei Zhang College of Information Science & Electronic Engineering Zhejiang University, China {jint zju, siyuhuang, yingming, zhongfei}@zju.edu.cn Abstract This paper addresses the challenging task of video captioning which aims to gener- ate descriptions for video data. Recently, the attention-based encoder-decoder structures have been widely used in video captioning. In existing literature, the attention weights are often built from the information of an indi- vidual modality, while, the association rela- tionships between multiple modalities are ne- glected. Motivated by this observation, we propose a video captioning model with High- Order Cross-Modal Attention (HOCA) where the attention weights are calculated based on the high-order correlation tensor to capture the frame-level cross-modal interaction of dif- ferent modalities sufficiently. Furthermore, we novelly introduce Low-Rank HOCA which adopts tensor decomposition to reduce the ex- tremely large space requirement of HOCA, leading to a practical and efficient implemen- tation in real-world applications. Experimen- tal results on two benchmark datasets, MSVD and MSR-VTT, show that Low-rank HOCA establishes a new state-of-the-art. 1 Introduction Video captioning has drawn much attention from natural language processing and computer vision researchers (Venugopalan et al., 2014; Bin et al., 2016; Ramanishka et al., 2016; Zanfir et al., 2016). As videos typically consist of multiple modalities (image, motion, audio, etc.), video cap- tioning is actually a multimodal learning task. The abundant information underlying in the modalities is much beneficial for video captioning models. However, how to effectively learn the association relationships between different modalities is still a challenging problem. In the context of deep learning based video captioning, multimodal attention mechanisms (Xu ∗Corresponding author. GT1: a referee coaches a wrestling match GT2: boys are wrestling in front of a crowd Figure 1: An example of video captioning, where ”GT” denotes ground truth. et al., 2017; Hori et al., 2017; Wang et al., 2018c) are shown to help deliver superior performance. Bahdanau Attention (Bahdanau et al., 2014) is widely used to calculate the attention weights ac- cording to the features of individual modalities, since Bahdanau Attention is originally proposed for machine translation which can be considered as a unimodal task. However, video captioning is a multimodal task and different modalities are able to provide complementary cues to each other when calculating the attention weights. For example, as shown in Fig. 1, a part of the video shows that two men are wrestling and another part shows that the referee coaches the match. With only the image modality, the model may pay equal attention to the frames with the competitors and referee. Consid- ering the additional motion modality, the attention mechanism weighs more on one of them, resulting in a model with more focused attention. Motivated by the above observations, in this paper …

Content truncated to 500 words