Hypothesizing Word Association From Untagged Text

HYPOTHESIZING UNTAGGED TEXT WORD ASSOCIATION FROM Tomoyoshi Matsukawa BBN Systems and Technologies 70 Fawcett St. Cambridge, MA 02138 ABSTRACT This paper reports a new method for suggesting word associations, based on a greedy algorithm that employs Chi- square statistics on joint frequencies of pairs of word groups compared against chance co-occurrence. The benefits of this new approach are: 1) we can consider even low frequency words and word pairs, and 2) word groups and word associations can be automatically generated. The method provided 87% accuracy in hypothesizing word associations for unobserved combinations of words in Japanese text. 1. INTRODUCTION Using mutual information for measuring word association has become popular since [Church and Hanks, 1990] defined word association ratio as mutual information between two words. Word association ratios are a promising tool for lexicography, but there seem to be at least two limitations to the method: 1) much data with low frequency words or word pairs cannot be used and 2) generalization of word usage still depends totally on lexicographers. In this paper, we propose an alternative (or extended) method for suggesting word associations using Chi-square statistics, which can be viewed as an approximation to mutual information. Rather than considering significance of joint frequencies of word pairs as [Church and Hanks, 1990] did, our algorithm uses joint frequencies of pairs of word groups instead. The algorithm employs a hill- climbing search for a pair of word groups that occur significantly frequently. The benefits of this new approach are: 1) that we can consider even low frequency words and word pairs, and 2) that word groups or word associations can be automatically generated, .namely automatic hypothesis of word associations, which can later be reviewed by a lexicographer. 3) word associations can be used in parsing and understanding natural language, as well as in natural language generation [Smadja and McKeown, 1990]. Our method proved to be 87% accurate in hypothesizing word associations for unobserved combinations of words in Japanese text, where accuracy was tested by human verification of a random sample of hypothesized word pairs. We extracted 14,407 observations of word co-occurrences, involving 3,195 nouns and 4,365 verb/argument pairs. Out of this we hypothesized 7,050 word associations. The corpus size was 280,000 words. We would like to apply the same approach to English. 2. RELATED WORK Some previous work (e.g., [Weischedel, et al., 1990]) found verb-argument associations from bracketed text, such as that in TREEBANK; however, this paper, and related work has hypothesized word associations from untagged text. [Hindle 1990] confirmed that word association ratios can be used for measuring similarity between nouns. For example, "ship", "plane", "bus", etc., were automatically ranked as similar to "boat". [Resnik 1992] reported a word association ratio for identifying noun classes from a pre- existing hierarchy as selectional constraints on the object of a verb. [Brown et.al. 1992] proves that, under the assumption of a bi-gram class model, the perplexity of a corpus is minimized when the average mutual information between word classes is maximized. Based on …

Content truncated to 500 words