GHAN: Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5547 - 5557 December 7-11, 2022 ©2022 Association for Computational Linguistics GHAN: Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval Yahan Yu1∗, Bojie Hu1 and Yu Li1,2 1 Tencent Minority-Mandarin Translation, Beijing, China 2 Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, China jasmineyuyh@gmail.com, bojiehu@tencent.com, 20112043@bjtu.com Abstract Text-video retrieval focuses on two aspects: cross-modality interaction and video-language encoding. Currently, the mainstream approach is to train a joint embedding space for multi- modal interactions. However, there are struc- tural and semantic differences between text and video, making this approach challenging for fine-grained understanding. In order to solve this, we propose an end-to-end graph-based hi- erarchical aggregation network for text-video retrieval according to the hierarchy possessed by text and video. We design a token-level weighted network to refine intra-modality rep- resentations and construct a graph-based mes- sage passing attention network for global-local alignment across modality. We conduct exper- iments on the public datasets MSR-VTT-9K, MSR-VTT-7K and MSVD, and achieve Re- call@1 of 73.0%, 65.6%, and 64.0% , which is 25.7%, 16.5%, and 14.2% better than the current state-of-the-art model. 1 Introduction Text-Video Retrieval (TVR) is a fundamental re- search task in multimodal video and language un- derstanding, which aims to retrieve the most rel- evant video for a given text query, or to retrieve relevant text for a given video query. The main- stream modeling approaches for TVR are to jointly learn cross-modal interactive information between video and text in the same representation space (Lei et al., 2021; Dzabraev et al., 2021; Luo et al., 2020; Liu et al., 2021; Cheng et al., 2021). However, the mismatched problem of informa- tion capacity and information density between video and text is not fully studied in these ap- proaches. As shown in Figure 1, there are structural and semantic differences between the modalities, making this approach challenging for TVR. The video itself expresses a much wider range of global ∗Work was done when Yahan Yu was interning at Minority-Mandarin Translation, Tencent Inc, China. Figure 1: Examples of differences between text and video. (a) The content mentioned in the text appears in a small number of frames in the video. (b) Text contains a different semantic range than video, making it difficult to achieve semantic alignment. representations than the text, so the textual seman- tic usually cannot be fully mapped to every detailed information of the video. If the content mentioned by the text appears in a small number of frames in video, the text may be misled by unrelated seman- tics when interacting across modalities. In addition, the local representations in text are usually more specific than that in video. Due to different annota- tors, text may have different description habits and contain different semantic ranges, such as using person/man/actor to describe a character appearing in the video, which makes it difficult with semantic understanding and alignment across modalities. If text and video are divided into multiple parts …

Content truncated to 500 words