Segmenting a Sentence Into Morphemes Using Statistic Information Between Words

SE(IMENTING A SENTENf,I¢ INTO MOItl)IIEM1,;S USING STNI'ISTIC INFOI{MATION BI,TFWEEN WORI)S Shiho Nobesawa Junya Tsutsumi, Tomoaki Nitta, Kotaro One, Sun Da Jiang, M~Lsakazu Nakanishi N;tkanishi L;d)oratory Faculty of Science and Technology, Keio University ABSTRACT This paper is on dividing non-separated language sen- tences (whose words are not separated from each other with a space or other separaters) into morphemes using statistical information, not grammatical infor- mation which is often used in NLP. In this paper we describe our method and experimental result on Japanese and Chinese se~,tences. As will be seen in the body of this paper, the result shows that this sys- tent is etlicient for most of tile sentences. 1 INTRODUCTION AND MOTIVATION An English sentence has several words and those words are separated with a space, it is e~usy to divide an English sentence into words. I[owever a a apalmse sen- tence needs parsing if you want to pick up the words in the sentence. This paper is on dividing non-separated language sentences into words(morphemes) without using any grammatical information. Instead, this sys- tem uses the statistic information between morphenws to select best ways of segmenting sentences in non- separated languages. Thinldng about segmenting a sentence into pieces, it is not very hard to divide a sentence using a cer- tain dictionary for that. The problem is how to de- cide which 'segmentation' the t)est answer is. For ex- aml)le , there must be several ways of segmenting a Japanese sentence written in lliragana(Jal)a,lese al- phabet). Maybe a lot more than 'several'. So, to make the segmenting system useful, we have to cot> sider how to pick up the right segmented sentences from all the possible seems-like-scgrne, nted sentences, This system is to use statistical inforn,ation be- tween morphemes to see how 'sentence-like'(how 'likely' to happen a.s a sentence) the se.gmented string is. To get the statistical association between words, mutual information(MI) comes to be one of the most inter- esting method. In this paper MI is used to calculate the relationship betwee.n words found ill the given sen- tence. A corpus of sentences is used to gain the MI. 'Fo implement this method, we iml)lemented a sys- tem MSS(Morphological Segmentation using Statisti- cal information). What MSS does is to find the best way of segmenting a non-separated language, sentence into morphemes without depending on granamatieal information. We can apply this system to many lan- guages. ~2 )/[ORPHOLOGICAL ANALYSIS 2.1 What; a Morphological Analysis Is A morpheme is the smallest refit of a string of char- acters which has a certain linguistic l/leaning itself. It includes both content words and flmction words, in this l)aper the definition of a morl)heme is a string of characters which is looked u I) in tile dictionary. Morphoh)gical analysis is to: l) recognize the smallest units making up tile given sentellce if the sentence is of a l|on-separated hmguage, divide the sentence into morphenms (auto- matic segmentation), and 2) check the morlflmmes whether they are the right units to make up the …

Content truncated to 500 words