Restructuring Tagged Corpora with Morpheme Adjustment Rules

Restructuring Tagged Corpora with Morpheme Adjustment Rules Abstract Toshihisa Tashiro Noriyoshi Uratani Tsuyoshi Morimoto ATR Interpreting Telecommunications Research Labor~tories 2-2 tIikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JAPAN {tashiro,uratani,morimoto} ~itl.at r.co.jp A part-of-speech tagged corpus is a very hnportant knowledge source for natural language processing researchers. 'Poday, several part-of-speech tagged corpora are readily available for research use. [Iowever, because there is wide diversity of morphological information systems (word-segmentation~ part-of-speech system, etc.), it is ditficult to use tagged corpora with an incompatible morphological infm'mation system. This paper proposes a me,hod of converting tagged corpora frOlll olte lllorphellle system to allother, 1 Introduction Recently, mm,y natural language processing re- searchers have concentrated on corpus-based ap- proaches. Linguistic corpora can be classified as word-segmented corpora, part-of-speech tagged cor- pora, and parsed corpora, tlecause a part-of-speech tagged corpus is the most important corpus, much corpus-based natural language processing research has been performed using part-of-speech tagged cor- pora. Ilowever, building a large part-of-speech tagged corpus is very dillicult. It is even more difficult to build a corpus for languages without explicit word boundary characters, such as Japanese. q'herefore, researchers always complain of the scarcity of data in the corpus. To solve this data scarcity problem, previous works proposed methods of increasing the productivity of the labor required for building a part-of-speech tagged corpus. [1]. '.l'his paper proposes another method of acquir- ing large part-of-speech tagged corpora: restructur- ing tagged corpora by using morpheme adjustment rules. This method assures good use of the sharable part-of-speech tagged corpora that are already awdl- able such as the ATR Dialog Database (ADD) [2; 3]. Ideally, these corpora could be used by all re- searchers and research groups without any modifica- tions, llowever, actual part-of-speech tagged corpora have the following problems: • Diversity of orthography: A word can be spelled in various ways. In Japanese, there are three types of character sets: kanji (i~.~"-), hiragana (~ to ~ ~), and katakana (:~J # :~J ¢). Also, people can use these character sets at their discretion. • Diversity of word segmentation: Because the Japanese language has no word boundary characters(i.e, blank spaces), there are no standards of word segmentation. A sin- gle word in a certain corpus may be considered nmltiple words in other corpora, and vice versa. • Diw;rsity of part-of speech systems: There are no standards for part-of-speech sys- tems. It is true that a detailed part-of-speech system can help the application of part-of-speech information, lint the labor required ['or lmilding corpora will continue to increase. This problem is language-independent. Diversities of word-segmentation and part-of- speech systems are fatal problems. The simplest way to solve these problems is to perform a morphological analysis on the raw text in the corpus, with no regard to the word-segmentation and part-of-speech infer mation. Uowever, making a high-quality morphologi- cal analyzer demands much time and care. Addition- ally, it is wasteful to ignore the word-segmentation and part-of-speech i,fformation that h~s been ac- quired with much effort. In restructuring tagged corpora with nmrpheme ad- justmel,t rules, …

Content truncated to 500 words