A Classification Tree Approach to Automatic Segmentation of Japanese Compound Sentences

A CLASSIFICATION TREE APPROACH TO AUTOMATIC SEGMENTATION OF JAPANESE COMPOUND SENTENCES Y ujie Zhang and Kazuhiko Ozeki Department of Computer Science and Information Mathematics, The University of Electro-Communications, Tokyo, Japan Email: zhang©achilleus.cs.uec.ac.jp, ozekiOcs.uec.ac.jp ABSTRACT It is well known that direct parsing of a long Japanese compound sentence is extremely difficult. Various pre-processing methods have been proposed to segment such a sentence into shorter, simpler ones prior to parsing. The problem with the conventional methods is that some kind of segmentation patterns or heuristic preference scores must be given man- ually, hence no guarantee for optimality. This paper proposes a new method of sentence segmentation based on a classification tree technique. In this method, optimal segmenta- tion patterns and the optimal order of their application are automatically acquired from training data, linguistic phenomena together with their occurence frequencies being taken into account. Generation of a classification tree is conducted on an EDR corpus, and eval- uation results are reported. It is shown that pruning reduces the tree size by a factor of about 1/4 without affecting the performance. 1. INTRODUCTION It is well known that direct parsing of a long Japanese compound sentence, comprising many coordinate clauses, is extremely difficult. Various pre-processing methods have been proposed to segment such a sentence into shorter, simpler ones prior to parsing [1]. Sentence segmentation have also been discussed from a view point of document revision support system [2], because a long compound sentence is difficult to understand even for humans. The techniques of sentence segmentation reported so far can be summarized as follows: (1) Segmentation points are estimated by matching between prescribed segmentation patterns and an input sentence. The segmentation patterns are described by using the part-of-speech and the orthographic representation of morphemes obtained by morphological analysis [1]. (2) Dependency analysis on a clause sequence is conducted based on subordination rela- tion among clauses [3]. Then dependency structure candidates are ordered by using heuristic dependency scores between clauses. Finally the segmentation points are determined in accordance with the top candidate for the dependency structure [2]. These techniques have been reported effective. However, the problem with these conven- tional methods is that the segmentation patterns or the heuristic dependency scores must be given manually, hence no guarantee for optimality. This paper proposes a new method of automatic segmentation of long compound sentences using a classification tree technique [4,5,6,7] based on the surface information obtained by morphological analysis. In this method, optimal segmentation patterns and the optimal order of their application are automatically acquired from training data, linguistic phenom- ena together with their occurrence frequencies being taken into account. The rest of the paper describes the details of the method, and reports the experimental results on an EDR corpus, including the effects of pruning. 2. CLASSIFICATION TREE The classification tree employed in this work is of the following type: (1) It is a binary tree: each intermediate node has two child-nodes. (2) Gini index [4] is employed for the measurement of impurity. (3) In the tree …

Content truncated to 500 words