Sentences vs. Phrases: Syntactic Complexity in Multimedia Information Retrieval

Sentences vs. Phrases: Syntactic Complexity in Multimedia Information Retrieval Sharon Flank emotion, Inc. 2600 Park Tower Dr., Ste. 600, Vienna, VA 22180 sharon.flank@emotion.corn Abstract In experiments on a natural language information retrieval system that retrieves images based on textual captions, we show that syntactic complexity actually aids retrieval. We compare two types of captioned images, those characterized with full sentences in English, and those characterized by lists of words and phrases. The full-sentence captions show a 15% increase in retrieval accuracy over the word- list captions. We conclude that the syntactic complexity may be of use in fact because it decreases semantic ambiguity: the word-list captions may be syntactically simple, but they are semantically confusingly complex. 1 Introduction In this paper, we describe experiments conducted on an image retrieval system, PictureQuest, which uses text captions to characterize images. The text captions are of two types. Optimally, they consist of a prose description of the image, generally two to three sentences, with perhaps three or four additional words or phrases that describe emotional or non-literal image content, e.g. Two little girls play with blocks. The younger girl, wearing a blue shirt, laughs and prepares to knock over the tower that the older girl has constructed The older girl, dressed in a red shirt, winces in anticipation. Siblings, cooperation, rivalry Some of the captions in PictureQuest are not as well-behaved. They may contain legacy data or data shared with a keyword-retrieval system. They are optimized for exact-match retrieval, and, as such, consist of lists of words or, at best, a few short phrases mixed in with long lists of words. The same image might appear with the following caption: girl, girls, little girl little girls, block, blocks, play, playing, plays, blue, red, shirt, tower, knock; over, construct, construction, siblings, cooperation, rivalry PictureQuest relies on several natural language processing techniques to enhance retrieval accuracy. It contains a part-of- speech tagger, morphological analyzer, noun phrase pattern matcher, semantic expansion based on WordNet, and special processing for names and locations. These have been tuned to perform most effectively on caption text of the first type, i.e. sentences. The following chart illustrates how these linguistic processes operate - or fail to operate - on syntactic units. Tagger dog-N herding-V sheep- N dog-N,V; herding-N,V; sheep-N Morphology dog herd-ING sheep (same) NP Pattems small child wearing a small, child, wearing, hat hat green, swirls (modifiers de-coupled from head green swirls nouns) cat jumping into the air: cat-N (7 senses) jumping-V (13 senses) air-N (13 senses) Semantic Expansion (WordNet- based) cat, jumping, air cat-N,V (9 senses) jumping-N,V,Adj (16 senses) air-N,V,Adj (20 senses) Names George Bush, A1 Gore George, Bush, AI, Gore (matches bush, gore) Locations Arlington, Virginia Arlington, Virginia (matches other Arlingtons in New England other states) New, England (matches England, new) 2 Complexity Measures 2.1 Competing Complexity Measures How do we determine what syntactic complexity is? Does it relate to depth? Nesting? Various definitions have been used in the various research communities: Alzheimer's research, normal and abnormal child …

Content truncated to 500 words