Dependency Structure Analysis and Sentence Boundary Detection in Spontaneous Japanese

Dependency Structure Analysis and Sentence Boundary Detection in Spontaneous Japanese Kazuya Shitaoka† Kiyotaka Uchimoto‡ Tatsuya Kawahara† Hitoshi Isahara‡ †School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto 606-8501, Japan, {shitaoka,kawahara}@ar.media.kyoto-u.ac.jp ‡National Institute of Information and Communications Technology 3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto 619-0289, Japan, {uchimoto,isahara}@nict.go.jp Abstract This paper describes a project to detect dependen- cies between Japanese phrasal units called bunsetsus, and sentence boundaries in a spontaneous speech corpus. In monologues, the biggest problem with de- pendency structure analysis is that sentence bound- aries are ambiguous. In this paper, we propose two methods for improving the accuracy of sentence boundary detection in spontaneous Japanese speech: One is based on statistical machine translation us- ing dependency information and the other is based on text chunking using SVM. An F-measure of 84.9 was achieved for the accuracy of sentence bound- ary detection by using the proposed methods. The accuracy of dependency structure analysis was also improved from 75.2% to 77.2% by using automat- ically detected sentence boundaries. The accuracy of dependency structure analysis and that of sen- tence boundary detection were also improved by in- teractively using both automatically detected depen- dency structures and sentence boundaries. 1 Introduction The “Spontaneous Speech: Corpus and Pro- cessing Technology” project has been sponsor- ing the construction of a large spontaneous Japanese speech corpus, Corpus of Spontaneous Japanese (CSJ) (Maekawa et al., 2000). The CSJ is the biggest spontaneous speech corpus in the world, and it is a collection of monologues and dialogues, the majority being monologues such as academic presentations. The CSJ in- cludes transcriptions of speeches as well as audio recordings. Approximately one tenth of the CSJ has been manually annotated with information about morphemes, sentence boundaries, depen- dency structures, discourse structures, and so on. The remaining nine tenths of the CSJ have been annotated semi-automatically. A fu- ture goal of the project is to extract sentence boundaries, dependency structures, and dis- course structures from the remaining transcrip- tions. This paper focuses on methods for au- tomatically detecting sentence boundaries and dependency structures in Japanese spoken text. In many cases, Japanese dependency struc- tures are defined in terms of the dependency relationships between Japanese phrasal units called bunsetsus. To define dependency rela- tionships between all bunsetsus in spontaneous speech, we need to define not only the depen- dency structures in all sentences but also the inter-sentential relationships, or, discourse re- lationships, between the sentences, as depen- dency relationships between bunsetsus. How- ever, it is difficult to define and detect discourse relationships between sentences because of sig- nificant inconsistencies in human annotations of discourse structures, especially with regard to spontaneous speech. We also need to know intra-sentential dependency structures in order to use the results of dependency structure anal- ysis for sentence compaction in automatic text summarization or case frame acquisition. Be- cause it is difficult to define discourse relation- ships between sentences, depending on the ac- tual application, it is usually enough to define and detect the dependency structure of each sentence. Therefore, the CSJ …

Content truncated to 500 words