Morphological Analysis of a Large Spontaneous Speech Corpus in Japanese

Morphological Analysis of a Large Spontaneous Speech Corpus in Japanese Kiyotaka Uchimoto† Chikashi Nobata† Atsushi Yamada† Satoshi Sekine‡ Hitoshi Isahara† †Communications Research Laboratory 3-5, Hikari-dai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {uchimoto,nova,ark,isahara}@crl.go.jp ‡New York University 715 Broadway, 7th floor New York, NY 10003, USA sekine@cs.nyu.edu Abstract This paper describes two methods for de- tecting word segments and their morpho- logical information in a Japanese sponta- neous speech corpus, and describes how to tag a large spontaneous speech corpus accurately by using the two methods. The first method is used to detect any type of word segments. The second method is used when there are several definitions for word segments and their POS categories, and when one type of word segments in- cludes another type of word segments. In this paper, we show that by using semi- automatic analysis we achieve a precision of better than 99% for detecting and tag- ging short words and 97% for long words; the two types of words that comprise the corpus. We also show that better accuracy is achieved by using both methods than by using only the first. 1 Introduction The “Spontaneous Speech: Corpus and Process- ing Technology” project is sponsoring the construc- tion of a large spontaneous Japanese speech corpus, Corpus of Spontaneous Japanese (CSJ) (Maekawa et al., 2000). The CSJ is a collection of mono- logues and dialogues, the majority being mono- logues such as academic presentations and simu- lated public speeches. Simulated public speeches are short speeches presented specifically for the cor- pus by paid non-professional speakers. The CSJ in- cludes transcriptions of the speeches as well as audio recordings of them. One of the goals of the project is to detect two types of word segments and cor- responding morphological information in the tran- scriptions. The two types of word segments were defined by the members of The National Institute for Japanese Language and are called short word and long word. The term short word approximates a dic- tionary item found in an ordinary Japanese dictio- nary, and long word represents various compounds. The length and part-of-speech (POS) of each are dif- ferent, and every short word is included in a long word, which is shorter than a Japanese phrasal unit, a bunsetsu. If all of the short words in the CSJ were detected, the number of the words would be approximately seven million. That would be the largest spontaneous speech corpus in the world. So far, approximately one tenth of the words have been manually detected, and morphological information such as POS category and inflection type have been assigned to them. Human annotators tagged every morpheme in the one tenth of the CSJ that has been tagged, and other annotators checked them. The hu- man annotators discussed their disagreements and resolved them. The accuracies of the manual tagging of short and long words in the one tenth of the CSJ were greater than 99.8% and 97%, respectively. The accuracies were evaluated by random sampling. As it took over two …

Content truncated to 500 words