Extracting Noun Phrases from Large-Scale Texts: A Hybrid Approach and Its Automatic Evaluation

Extracting Noun Phrases from Large-Scale Texts: A Hybrid Approach and Its Automatic Evaluation Kuang-hua Chen and Hsin-Hsi Chen Department of Computer Science and Information Engineering National Taiwan University Taipei, Taiwan, R.O.C. Internet: hh_chen@csie, ntu. edu.tw Abstract To acquire noun phrases from running texts is useful for many applications, such as word grouping, terminology indexing, etc. The reported literatures adopt pure probabilistic approach, or pure rule-based noun phrases a grammar to tackle this problem. In this paper, we apply probabilistic chunker to deciding the implicit boundaries of constituents and utilize the linguistic knowledge to extract the noun phrases by a finite state mechanism. The test texts are SUSANNE Corpus and the results are evaluated by comparing the parse field of SUSANNE Corpus automatically. The results of this preliminary experiment are encouraging. 1. Introduction From the cognitive point of view, human being must recognize, learn and understand the entities or concepts (concrete or abstract) in the texts for natural language comprehension. These entities or concepts are usually described by noun phrases. The evidences from the language learning of children also show the belief (Snow and Ferguson, 1977). Therefore, if we can grasp the noun phases of the texts, we will understand the texts to some extent. This consideration is also captured by theories of discourse analysis, such as Discourse Representation Theory (Kamp, 1981). Traditionally, to make out the noun phrases in a text means to parse the text and to resolve the attachment relations among the constituents. However, parsing the text completely is very difficult, since various ambiguities cannot be resolved solely by syntactic or semantic information. Do we really need to fully parse the texts in every application? Some researchers apply shallow or partial parsers (Smadja, 1991; Hindle, 1990) to acquiring specific patterns from texts. These tell us that it is not necessary to completely parse the texts for some applications. This paper will propose a probabilistic partial parser and incorporate linguistic knowledge to extract noun phrases. The partial parser is motivated by an intuition (Abney, 1991): (1) When we read a sentence, we read it chunk by chunk. Abney uses two level grammar rules to implement the parser through pure LR parsing technique. The first level grammar rule takes care of the chunking process. The second level grammar rule tackles the attachment problems among chunks. Historically, our statistics- based partial parser is called chunker. The chunker receives tagged texts and outputs a linear chunk sequences. We assign a syntactic head and a semantic head to each chunk. Then, we extract the plausible maximal noun phrases according to the information of syntactic head and semantic head, and a finite state mechanism with only 8 states. Section 2 will give a brief review of the works for the acquisition of noun phrases. Section 3 will describe the language model for chunker. Section 4 will specify how to apply linguistic knowledge to assigning heads to each chunk. Section 5 will list the experimental results of chunker. Following Section 5, Section 6 will give …

Content truncated to 500 words