Toward Multimedia: A String Pattern-Based Passage Ranking Model for Video Question Answering

Proceedings of NAACL HLT 2007, pages 540–547, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Toward Multimedia: A String Pattern-based Passage Ranking Model for Video Question Answering Yu-Chieh Wu Jie-Chi Yang Dept. of Computer Science and Infor- mation Engineering Graduate Institute of Network Learning Technology National Central University National Central University Taoyuan, Taiwan Taoyuan, Taiwan bcbb@db.csie.ncu.edu.tw yang@cl.ncu.edu.tw Abstract In this paper, we present a new string pat- tern matching-based passage ranking al- gorithm for extending traditional text- based QA toward videoQA. Users interact with our videoQA system through natural language questions, while our system re- turns passage fragments with correspond- ing video clips as answers. We collect 75.6 hours videos and 253 Chinese ques- tions for evaluation. The experimental re- sults showed that our method outperformed six top-performed ranking models. It is 10.16% better than the sec- ond best method (language model) in rela- tively MRR score and 6.12% in precision rate. Besides, we also show that the use of a trained Chinese word segmentation tool did decrease the overall videoQA per- formance where most ranking algorithms dropped at least 10% in relatively MRR, precision, and answer pattern recall rates. 1 Introduction With the drastic growth of video sources, effective indexing and retrieving video contents has recently been addressed. The well-known Informedia pro- ject (Wactlar, 2000) and TREC-VID track (Over et al., 2005) are the two famous examples. Although text-based question answering (QA) has become a key research issue in past decade, to support mul- timedia such as video, it is still beginning. Over the past five years, several video QA stud- ies had investigated. Lin et al. (2001) presented an earlier work on combining videoOCR and term weighting models. Yang et al. (2003) proposed a complex videoQA approach by employing abun- dant external knowledge such as, Web, WordNet, shallow parsers, named entity taggers, and human- made rules. They adopted the term-weighting method (Pasca, and Harabagiu, 2001) to rank the video segments by weighting the pre-defined key- words. Cao and Nunamaker (2004) developed a lexical pattern matching-based ranking method for a domain-specific videoQA. In the same year, Wu et al. (2004) designed a cross-language (English- to-Chinese) video question answering system based on extracting pre-defined named entity words in captions. On the other hand, Zhang and Nunamaker (2004) made use of the simple TFIDF term weighting schema to retrieve the manual- segmented clips for video caption word retrieval. They also manually developed the ontology to im- prove system performance. In this paper, we present a new string pattern matching-based passage ranking algorithm for video question answering. We consider that the passage is able to answer questions and also suit- able for videos because itself forms a very natural unit. Lin et al. (2003) showed that users prefer pas- sage-level answers over short answer phrases since it contains rich context information. Our method makes use of the string pattern searching in the suffix trees to find common subsequences between a passage and question. The proposed term weight- ing schema is then …

Content truncated to 500 words