Overview of Natural Language Processing of Captions for Retrieving Multimedia Data

Overview of Natural Language Processing of Captions for Retrieving Multimedia Data Eugene J. Guglielmo Naval Weapons Center Information Systems Department, Code 2724 China Lake, California 93555 Neil C. Rowe Naval Postgraduate School Computer Science Department, Code 52 Monterey, California 93943 Abstract This paper briefly describes the current implementa- tion status of an intelligent information retrieval sys- tem, MARIE, that employs natural language process- ing techniques. Descriptive captions are used to iden- tify photographic images concerning various military projects. The captions are parsed to produce a logical form from which nouns and verbs are extracted to form the primary keywords. User queries are also specified in natural language. A two-phase search pro- cess employing coarse-grain and fine-grain match pro- cesses is used to find the captions that best match the query. A type hierarchy based on object-oriented pro- gramming constructs is used to represent the semantic knowledge base. This knowledge base contains knowledge of various military concepts and terminol- ogy with specifics from the Naval Weapons Center. Methods are used for creating the logical form during semantic analysis, generating the keywords to be used in the coarse-grain match process, and fine-grain matching between query and caption logical forms. 1 Introduction Recent approaches to intelligent information retrieval have used natural language (NL) understanding methods instead of keywords and statistical methods. However, the best NL method is still unknown. This research studies a restricted form of information, the description associated with identifying multimedia data, i.e., natural language captions. The rationale and motivation for using captions was presented by Lum and Meyer-Wegener (1990). A prototype parser was developed for demonstrating how natural language queries could be used in conjunction with Structured Query Language (SQL) for specifying retrieval requests from a multimedia database. Using these results at the Naval Postgraduate School, we have been able to design a more robust natural language processing and retrieval system for potential use at the Naval Weapons Center (NWC). The Center's Photo Lab maintains a database of over 100,000 photographs of project and historical data from the last 50 years. Both captions and supercaptions (caption about a set of captions) are used. The current search and retrieval strategy uses manually created keywords organized into a keyphrase - a head keyword and a string of descriptive nouns. Our strategy entails parsing the English captions to produce a logical form, then using the logical form as the basis of the retrieval. We have labeled this system MARIE (Epistemological Information Retrieval Applied to Multimedia). 2 Methodology The information retrieval system we have developed is based on two stages:.a coarse-grain match to reduce the list of possible information for a later fine-grain match (Rau 1987). Three tasks that we deemed essential for this system included the ability to represent and produce a logical form of the caption, the ability to generate keywords from the logical form, and the ability to load in previously stored caption logical forms for matching against the query logical form. 2.1 NL Parser We have used an existing natural …

Content truncated to 500 words