Automatic Identification of Non-compositional Phrases

Automatic Identification of Non-compositional Phrases Dekang Lin Department of Computer Science University of Manitoba and Winnipeg, Manitoba, Canada, R3T 2N2 lindek@cs.umanitoba.ca UMIACS University of Maryland College Park, Maryland, 20742 lindek@umiacs.umd.edu Abstract Non-compositional expressions present a special challenge to NLP applications. We present a method for automatic identification of non-compositional ex- pressions using their statistical properties in a text corpus. Our method is based on the hypothesis that when a phrase is non-composition, its mutual infor- mation differs significantly from the mutual infor- mations of phrases obtained by substituting one of the word in the phrase with a similar word. 1 Introduction Non-compositional expressions present a special challenge to NLP applications. In machine transla- tion, word-for-word translation of non-compositional expressions can result in very misleading (sometimes laughable) translations. In information retrieval, ex- pansion of words in a non-compositional expression can lead to dramatic decrease in precision without any gain in recall. Less obviously, non-compositional expressions need to be treated differently than other phrases in many statistical or corpus-based NLP methods. For example, an underlying assumption in some word sense disambiguation systems, e.g., (Da- gan and Itai, 1994; Li et al., 1995; Lin, 1997), is that if two words occurred in the same context, they are probably similar. Suppose we want to determine the intended meaning of "product" in "hot product". We can find other words that are also modified by "hot" (e.g., "hot car") and then choose the mean- ing of "product" that is most similar to meanings of these words. However, this method fails when non-compositional expressions are involved. For in- stance, using the same algorithm to determine the meaning of "line" in "hot line", the words "product", "merchandise", "car", etc., would lead the algorithm to choose the "line of product" sense of "line". We present a method for automatic identification of non-compositional expressions using their statis- tical properties in a text corpus. The intuitive idea behind the method is that the metaphorical usage of a non-compositional expression causes it to have a different distributional characteristic than expres- sions that are similar to its literal meaning. 2 Input Data The input to our algorithm is a collocation database and a thesaurus. We briefly describe the process of obtaining this input. More details about the con- struction of the collocation database and the the- saurus can be found in (Lin, 1998). We parsed a 125-million word newspaper corpus with Minipar, 1 a descendent of Principar (Lin, 1993; Lin, 1994), and extracted dependency relationships from the parsed corpus. A dependency relationship is a triple: (head type modifier), where head and modifier are words in the input sentence and type is the type of the dependency relation. For example, (la) is an example dependency tree and the set of dependency triples extracted from (la) are shown in (lb). compl John married Peter's sister b. (marry V:subj:N John), (marry V:compl:N sister), (sister N:gen:N Peter) There are about 80 million dependency relation- ships in the parsed corpus. The frequency counts of dependency relationships are filtered …

Content truncated to 500 words