Using Visual Information to Predict Lexical Preference

Proceedings of Recent Advances in Natural Language Processing, pages 399–405, Hissar, Bulgaria, 12-14 September 2011. Using Visual Information to Predict Lexical Preference Shane Bergsma Dept. of Computer Science and HLTCOE Johns Hopkins University sbergsma@jhu.edu Randy Goebel Dept. of Computing Science University of Alberta goebel@cs.ualberta.ca Abstract Most NLP systems make predictions based solely on linguistic (textual or spo- ken) input. We show how to use visual information to make better linguistic pre- dictions. We focus on selectional prefer- ence; specifically, determining the plau- sible noun arguments for particular verb predicates. For each argument noun, we extract visual features from corresponding images on the web. For each verb predi- cate, we train a classifier to select the vi- sual features that are indicative of its pre- ferred arguments. We show that for certain verbs, using visual information can signif- icantly improve performance over a base- line. For the successful cases, visual infor- mation is useful even in the presence of co- occurrence information derived from web- scale text. We assess a variety of training configurations, which vary over classes of visual features, methods of image acquisi- tion, and numbers of images. 1 Introduction Selectional preferences quantify the plausibility of predicate-argument pairs. We focus on pre- dicting the plausibility of a noun argument (e.g. pasta) occurring as the direct object of a verb predicate (e.g. eat). Such knowledge is useful since many NLP tasks require determining the ac- tual argument from the alternatives that arise be- cause of syntactic, semantic or anaphoric ambi- guity. Previous uses of selectional preferences include prepositional-phrase attachment (Hindle and Rooth, 1993), word-sense disambiguation (Resnik, 1997), pronoun resolution (Dagan and Itai, 1990), and semantic role labeling (Erk, 2007). The compatibility of a predicate and an argu- ment can be quantified by counting how often they occur together in a large text corpus (Hindle and Rooth, 1993), but many plausible pairs are absent even from web-scale text (Bergsma et al., 2008). We therefore seek to generalize from observed pairs in order to make inferences for unseen com- binations. Some approaches back off to counts over argument classes (Resnik, 1996; Rooth et al., 1999; Clark and Weir, 2002; ´O S´eaghdha, 2010; Ritter et al., 2010), Others interpolate over simi- lar words (Dagan et al., 1999; Erk, 2007). Text- based approaches work best for arguments that are frequent in text, but, paradoxically, frequent argu- ments are the arguments for which generalization is least needed. This provides motivation to look beyond text in order to make better predictions for infrequent or out-of-vocabulary arguments. We propose using visual features to identify a verb’s preferred arguments. Visual information may play a role in the human acquisition of word meaning (Feng and Lapata, 2010b). For com- puters, there is a massive amount of visual data to exploit. Billions of images are added to web- sites like Facebook and Flickr every month. The challenge of associating words and images is re- duced because many users label their images as they post them online, providing an explicit link …

Content truncated to 500 words