Automatically Extracting and Representing Collocations for Language Generation

AUTOMATICALLY EXTRACTING AND REPRESENTING COLLOCATIONS FOR LANGUAGE GENERATION* Frank A. Smadja t and Kathleen R. McKeown Department of Computer Science Columbia University New York, NY 10027 ABSTRACT Collocational knowledge is necessary for language gener- ation. The problem is that collocations come in a large variety of forms. They can involve two, three or more words, these words can be of different syntactic cate- gories and they can be involved in more or less rigid ways. This leads to two main difficulties: collocational knowledge has to be acquired and it must be represented flexibly so that it can be used for language generation. We address both problems in this paper, focusing on the acquisition problem. We describe a program, Xtract, that automatically acquires a range of collocations from large textual corpora and we describe how they can be represented in a flexible lexicon using a unification based formalism. 1 INTRODUCTION Language generation research on lexical choice has fo- cused on syntactic and semantic constraints on word choice and word ordering. Colloca~ional constraints, however, also play a role in how words can co-occur in the same sentence. Often, the use of one word in a par- ticular context of meaning will require the use of one or more other words in the same sentence. While phrasal lexicons, in which lexical associations are pre-encoded (e.g., [Kukich 83], [Jacobs 85], [Danlos 87]), allow for the treatment of certain types of collocations, they also have problems. Phrasal entries must be compiled by hand which is both expensive and incomplete. Furthermore, phrasal entries tend to capture rather rigid, idiomatic expressions. In contrast, collocations vary tremendously in the number of words involved, in the syntactic cat- egories of the words, in the syntactic relations between the words, and in how rigidly the individual words are used together. For example, in some cases, the words of a collocation must be adjacent, while in others they can be separated by a varying number of other words. *The research reported in this paper was partially sup- ported by DARPA grant N00039-84-C-0165, by NSF grant IRT-84-51438 and by ONR grant N00014-89-J-1782. tMost of this work is also done in collaboration with Bell Communication Research, 445 South Street, Morristown, NJ 07960-1910 In this paper, we identify a range of collocations that are necessary for language generation, including open compounds of two or more words, predicative relations (e.g., subject-verb), and phrasal templates represent- ing more idiomatic expressions. We then describe how Xtract automatically acquires the full range of colloca- tions using a two stage statistical analysis of large do- main specific corpora. Finally, we show how collocations can be efficiently represented in a flexible lexicon using a unification based formalism. This is a word based lexicon that has been macrocoded with collocational knowledge. Unlike a purely phrasal lexicon, we thus retain the flexi- bility of word based lexicons which allows for collocations to be combined and merged in syntactically acceptable ways with other words or phrases of the sentence. Unlike pure word based …

Content truncated to 500 words