Rethinking Multi-Modal Alignment in Multi-Choice VideoQA from Feature and Sample Perspectives

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8188 - 8198 December 7-11, 2022 ©2022 Association for Computational Linguistics Rethinking Multi-Modal Alignment in Multi-Choice VideoQA from Feature and Sample Perspectives Shaoning Xiao♡, Long Chen♠∗, Kaifeng Gao♡, Zhao Wang♡, Yi Yang♡, Zhimeng Zhang♡, and Jun Xiao♡ ♡Zhejiang University ♠Columbia University {shaoningx,kite_phone,zhao_wang,yangyics,zhimeng,junx}@zju.edu.cn, zjuchenlong@gmail.com Abstract Reasoning about causal and temporal event re- lations in videos is a new destination of Video Question Answering (VideoQA). The major stumbling block to achieve this purpose is the semantic gap between language and video since they are at different levels of abstrac- tion. Existing efforts mainly focus on design- ing sophisticated architectures while utilizing frame- or object-level visual representations. In this paper, we reconsider the multi-modal alignment in VideoQA from feature and sam- ple perspectives to achieve better performance. From the view of feature, we break down the video into trajectories and first leverage trajec- tory feature in VideoQA to enhance the align- ment between two modalities. Moreover, we adopt a heterogeneous graph architecture and design a hierarchical framework to align both trajectory-level and frame-level visual feature with language feature. In addition, we found that VideoQA models are largely dependent on language priors and always neglect visual- language interactions. Thus, two effective yet portable training augmentation strategies are designed to strengthen the cross-modal corre- spondence ability of our model from the view of sample. Extensive results show that our method outperforms all state-of-the-art mod- els on the challenging NExT-QA benchmark. 1 Introduction Given a video and a question about its content, Video Question Answering (VideoQA) aims to an- swer the question through multi-modal reasoning. Since it can benefit numerous multi-modal appli- cations such as video retrieval (Zhang et al., 2021; Xiao et al., 2021c,b) and interaction with robot vi- sion, VideoQA has received increasing attention in recent years. Compared to Image Question An- swering (ImageQA) (Antol et al., 2015), VideoQA requires more complex reasoning. A naive exten- sion of ImageQA methods (Chen et al., 2020, 2022) ∗Long Chen is the corresponding author. … objects trajectories Q: Why did the boy in orange hold a ball on his head? A: want to throw the ball. Figure 1: An illustration of VideoQA. Existing VideoQA models utilize objects as finer clues. We fur- ther track the objects with same classes across time and first leverage video trajectory for better alignment. may not apply to the problem of VideoQA due to the extra information such as temporal object inter- actions. As computing systems are more frequently inter- vening to improve people’s work and daily lives, it is critical for machines to correctly comprehend the complex relationships between events such as causal and temporal relations. However, conven- tional VideoQA benchmarks (Xu et al., 2017; Yu et al., 2019) are constituted by descriptive ques- tions, e.g., asking the number, color or single ac- tion of the video elements. Recently, a benchmark called NExT-QA (Xiao et al., 2021a) was elabo- rately designed to challenge VideoQA models to reason about …

Content truncated to 500 words