Development of a Partially Bracketed Corpus with Part-of-Speech Information Only

Development of a Partially Bracketed Corpus with Part-of-Speech Information Only Hsin-Hsi Chen and Yue-Shi Lee Department of Computer Science and Information Engineering National Taiwan University Taipei, Taiwan, R.O.C. E-mail: hh_chen@csie.ntu.edu.tw Abstract Resea/ch based on a treebank is active for many natural language applications. However, the work to build a large scale treebank is laborious and tedious. This paper proposes a probabilistic chunker to help the development of a partially bracketed corpus. The chunker partitions the part-of-speech sequence into segments called chunks. Rather than using a treebank as our training corpus, a corpus which is tagged with part-of-speech information only is used. The experimental results show the probabilistic chunker has more than 92% correct rate in outside test. The well-formed partially bracketed corpus is a milestone in the development of a treebank. Besides, the simple but effective chunker can also be applied to many natural language applications. 1. Introduction Research based on a treebank, i.e., a corpus annotated with syntactic structures, is active for many natural language applications [1-5]. Framis [1] proposes a methodology to extract selectional restrictions at a variable level of abstraction from the Penn Treebank. Chen and Chen [2] propose a probabilistic chunker to decide the implicit boundaries of constituents and utilize the linguistic knowledge to extract the noun phrases by a finite state mechanism. In their study, Susanne Corpus is used as a trainmg corpus for their chunker. Pocock and Atwell [3] investigate statistical grammars extracted from Spoken English Corpus (SEC), and apply these grammars to find the grammatically optimal path through a word lattice. The stochastic parsers are also developed in [4,5]. All these applications employ the syntactic information extracted from different treebanks and show the satisfactory results. However, the work to build a large scale treebank is laborious and tedious. Very few large-scale treebanks are currently available especially for languages other than English. In this paper, we propose a probabilistic chunker to help the development of a partially bracketed corpus, i.e., a simpler version of a treebank. The chunker partitions the part-of-speech sequence into segments called chunks. Rather than using a treebank as our training corpus, a corpus which is tagged with part-of-speech information only is used. In the following sections we first introduce the experimental framework of our model. Lancaster- Oslo/Bergen (LOB) Corpus and Susanne Corpus are adopted. Then a tag mapper and a probabilistic chunker are described. Before concluding the experimental results are demonstrated. 2. Experimental Framework Because the probabilistic chunker proposed in this paper is based on syntactic tags (parts of speech), a part-of-speech tagger is needed. A word sequence W is input to the part-of-speech tagger and a part-of- 162 speech sequence P is generated. The output of the tagger is the input of the chunker. The probabilistic chunker partitions P into C, i.e., a sequence of chunks. Each chunk contains one or more parts of speech. Consider the example "Attorneys for the mayor said that an amicable property settlement has been agreed upon .". This 15-word sentence is input to …

Content truncated to 500 words