Evaluating Smoothing Algorithms against Plausibility Judgements

Evaluating Smoothing Algorithms against Plausibility Judgements Maria Lapata and Frank Keller Department of Computational Linguistics Saarland University PO Box 15 11 50 66041 Saarbr¨ucken, Germany {mlap, keller}@coli.uni-sb.de Scott McDonald Language Technology Group University of Edinburgh 2 Buccleuch Place Edinburgh EH8 9LW, UK scottm@cogsci.ed.ac.uk Abstract Previous research has shown that the plausibility of an adjective-noun com- bination is correlated with its corpus co-occurrence frequency. In this paper, we estimate the co-occurrence frequen- cies of adjective-noun pairs that fail to occur in a 100 million word corpus using smoothing techniques and com- pare them to human plausibility rat- ings. Both class-based smoothing and distance-weighted averaging yield fre- quency estimates that are significant predictors of rated plausibility, which provides independent evidence for the validity of these smoothing techniques. 1 Introduction Certain combinations of adjectives and nouns are perceived as more plausible than others. A classi- cal example is strong tea, which is highly plausi- ble, as opposed to powerful tea, which is not. On the other hand, powerful car is highly plausible, whereas strong car is less plausible. It has been argued in the theoretical literature that the plausi- bility of an adjective-noun pair is largely a collo- cational (i.e., idiosyncratic) property, in contrast to verb-object or noun-noun plausibility, which is more predictable (Cruse, 1986; Smadja, 1991). The collocational hypothesis has recently been investigated in a corpus study by Lapata et al. (1999). This study investigated potential statistical predictors of adjective-noun plausibility by using correlation analysis to com- pare judgements elicited from human subjects with five corpus-derived measures: co-occurrence frequency of the adjective-noun pair, noun frequency, conditional probability of the noun given the adjective, the log-likelihood ratio, and Resnik’s (1993) selectional association measure. All predictors but one were positively correlated with plausibility; the highest correlation was obtained with co-occurrence frequency. Resnik’s selectional association measure surprisingly yielded a significant negative correlation with judged plausibility. These results suggest that the best predictor of whether an adjective-noun combination is plausible or not is simply how often the adjective and the noun collocate in a record of language experience. As a predictor of plausibility, co-occurrence frequency has the obvious limitation that it can- not be applied to adjective-noun pairs that never occur in the corpus. A zero co-occurrence count might be due to insufficient evidence or might reflect the fact that the adjective-noun pair is in- herently implausible. In the present paper, we ad- dress this problem by using smoothing techniques (distance-weighted averaging and class-based smoothing) to recreate missing co-occurrence counts, which we then compare to plausibility judgements elicited from human subjects. By demonstrating a correlation between recreated frequencies and plausibility judgements, we show that these smoothing methods produce realistic frequency estimates for missing co-occurrence data. This approach allows us to establish the va- lidity of smoothing methods independent from a specific natural language processing task. 2 Smoothing Methods Smoothing techniques have been used in a variety of statistical natural language processing applica- tions as a means to address data sparseness, an in- herent problem for statistical …

Content truncated to 500 words