Annotation-Based Multimedia Summarization and Translation

Annotation-Based Multimedia Summarization and Translation Katashi Nagao Dept. of Information Engineering Nagoya University and CREST, JST Furo-cho, Chikusa-ku, Nagoya 464-8603, Japan nagao@nuie.nagoya-u.ac.jp Shigeki Ohira School of Science and Engineering Waseda University 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan ohira@shirai.info.waseda.ac.jp Mitsuhiro Yoneoka Dept. of Computer Science Tokyo Institute of Technology 2-12-1 O-okayama, Meguro-ku, Tokyo 152-8552, Japan yoneoka@img.cs.titech.ac.jp Abstract This paper presents techniques for multimedia annotation and their application to video sum- marization and translation. Our tool for anno- tation allows users to easily create annotation including voice transcripts, video scene descrip- tions, and visual/auditory object descriptions. The module for voice transcription is capable of multilingual spoken language identification and recognition. A video scene description con- sists of semi-automatically detected keyframes of each scene in a video clip and time codes of scenes. A visual object description is created by tracking and interactive naming of people and objects in video scenes. The text data in the multimedia annotation are syntactically and semantically structured using linguistic annota- tion. The proposed multimedia summarization works upon a multimodal document that con- sists of a video, keyframes of scenes, and tran- scripts of the scenes. The multimedia transla- tion automatically generates several versions of multimedia content in different languages. 1 Introduction Multimedia content such as digital video is be- coming a prevalent information source. Since the volume of such content is growing to huge numbers of hours, summarization is required to effectively browse video segments in a short time without missing significant content. Annotating multimedia content with semantic information such as scene/segment structures and metadata about visual/auditory objects is necessary for advanced multimedia content services. Since natural language text such as a voice transcript is highly manageable, speech and natural lan- guage processing techniques have an essential role in our multimedia annotation. We have developed techniques for semi- automatic video annotation integrating a mul- tilingual voice transcription method, some video analysis methods, and an interactive visual/auditory annotation method. The video analysis methods include automatic color change detection, characterization of frames, and scene recognition using similarity between frame attributes. There are related approaches to video annota- tion. For example, MPEG-7 is an effort within the Moving Picture Experts Group (MPEG) of ISO/IEC that is dealing with multimedia con- tent description (MPEG, 2002). MPEG-7 can describe indeces, notes, and so on, to retrieve necessary parts of content speedily. However, it takes a high cost to add these descriptions by hands. The method of extracting them auto- matically through the video/audio analysis is vitally important. Our method can be inte- grated into tools for authoring MPEG-7 data. The linguistic description scheme, which will be a part of the amendment to MPEG-7, should play a major role in this integration. Using such annotation data, we have also de- veloped a system for advanced multimedia pro- cessing such as video summarization and trans- lation. Our video summary is not just a shorter version of the original video clip, but an in- teractive multimedia presentation that shows keyframes of important scenes and their …

Content truncated to 500 words