Improving the Use of Pseudo-Words for Evaluating Selectional Preferences

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 445–453, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Improving the Use of Pseudo-Words for Evaluating Selectional Preferences Nathanael Chambers and Dan Jurafsky Department of Computer Science Stanford University {natec,jurafsky}@stanford.edu Abstract This paper improves the use of pseudo- words as an evaluation framework for selectional preferences. While pseudo- words originally evaluated word sense disambiguation, they are now commonly used to evaluate selectional preferences. A selectional preference model ranks a set of possible arguments for a verb by their se- mantic fit to the verb. Pseudo-words serve as a proxy evaluation for these decisions. The evaluation takes an argument of a verb like drive (e.g. car), pairs it with an al- ternative word (e.g. car/rock), and asks a model to identify the original. This pa- per studies two main aspects of pseudo- word creation that affect performance re- sults. (1) Pseudo-word evaluations often evaluate only a subset of the words. We show that selectional preferences should instead be evaluated on the data in its en- tirety. (2) Different approaches to select- ing partner words can produce overly op- timistic evaluations. We offer suggestions to address these factors and present a sim- ple baseline that outperforms the state-of- the-art by 13% absolute on a newspaper domain. 1 Introduction For many natural language processing (NLP) tasks, particularly those involving meaning, cre- ating labeled test data is difficult or expensive. One way to mitigate this problem is with pseudo- words, a method for automatically creating test corpora without human labeling, originally pro- posed for word sense disambiguation (Gale et al., 1992; Schutze, 1992). While pseudo-words are now less often used for word sense disambigation, they are a common way to evaluate selectional preferences, models that measure the strength of association between a predicate and its argument filler, e.g., that the noun lunch is a likely object of eat. Selectional preferences are useful for NLP tasks such as parsing and semantic role labeling (Zapirain et al., 2009). Since evaluating them in isolation is difficult without labeled data, pseudo- word evaluations can be an attractive evaluation framework. Pseudo-word evaluations are currently used to evaluate a variety of language modeling tasks (Erk, 2007; Bergsma et al., 2008). However, evaluation design varies across research groups. This paper studies the evaluation itself, showing how choices can lead to overly optimistic results if the evaluation is not designed carefully. We show in this paper that current methods of apply- ing pseudo-words to selectional preferences vary greatly, and suggest improvements. A pseudo-word is the concatenation of two words (e.g. house/car). One word is the orig- inal in a document, and the second is the con- founder. Consider the following example of ap- plying pseudo-words to the selectional restrictions of the verb focus: Original: This story focuses on the campaign. Test: This story/part focuses on the campaign/meeting. In the original sentence, focus has two arguments: a subject story and an object campaign. In the test sentence, each …

Content truncated to 500 words