Video Event Detection by Exploiting Word Dependencies from Image Captions

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3318–3327, Osaka, Japan, December 11-17 2016. Video Event Detection by Exploiting Word Dependencies from Image Captions Sang Phan†, Yusuke Miyao†, Duy-Dinh Le†§, Shin’ichi Satoh† †National Institute of Informatics, Japan §Multimedia Communications Lab, University of Information Technology, Vietnam {plsang,yusuke,ledduy,satoh}@nii.ac.jp Abstract Video event detection is a challenging problem in information and multimedia retrieval. Differ- ent from single action detection, event detection requires a richer level of semantic information from video. In order to overcome this challenge, existing solutions often represent videos using high level features such as concepts. However, concept-based representation can be confusing because it does not encode the relationship between concepts. This issue can be addressed by exploiting the co-occurrences of the concepts, however, it often leads to a very huge number of possible combinations. In this paper, we propose a new approach to obtain the relationship be- tween concepts by exploiting the syntactic dependencies between words in the image captions. The main advantage of this approach is that it significantly reduces the number of informative combinations between concepts. We conduct extensive experiments to analyze the effectiveness of using the new dependency representation for event detection on two large-scale TRECVID Multimedia Event Detection 2013 and 2014 datasets. Experimental results show that i) Depen- dency features are more discriminative than concept-based features. ii) Dependency features can be combined with our current event detection system to further improve the performance. For instance, the relative improvement can be as far as 8.6% on the MEDTEST14 10Ex setting. 1 Introduction Detecting event from videos has been an important research topic due to the explosion of internet videos. In order to build a reliable event detection system, one must rely on the video content rather than sim- ply using textual metadata (Davidson et al., 2010). However, internet videos are often captured under arbitrary conditions, which makes the large content variation among videos of the same event. To handle this problem, we can represent videos using multimodal features such as Dense trajectories (Wang and Schmid, 2013), SIFT (Lowe, 2004), and audio MFCC (Lee et al., 1988). Another approach to tackle this problem is to represent videos using the concept-based representation, i.e., video is represented by the concept detection scores indicating the presence of the concepts. This approach is particularly helpful when the number of training videos are scarce (Chen et al., 2014; Habibian et al., 2014b; Habibian et al., 2013; Ma et al., 2013; Ye et al., 2015). Some other works further select a subset of informative concepts for event detection (Jiang et al., 2015; Mazloom et al., 2013). Nevertheless, the concept-based representation has two drawbacks i) It is non-trivial to obtain a large concept vocabulary. That is why people often combine concepts from multiple vocabularies to construct a larger one, in the hope to cover a wide range of concepts in real world videos (Yu et al., 2014). ii) The relationship between concepts, e.g., co-occurrence, are not captured. However, these relationships …

Content truncated to 500 words