Morphemes as Necessary Concept for Structures Discovery from Untagged Corpora

m m m m / | m m m / / | l m m m Morphemes as Necessary Concept for Structures Discovery from Untagged Corpora Hervd D~jean GREYC - CNRS - UPRESA 6072 Universit~ de Caen - Basse Normandie Herve. Dej ean@inf o. unicaen, fr Abstract This paper describes an overview of a method which allows discovery of syntactic structures from untagged corpora. It is composed of three main steps: the discovery of the grammatical morphemes of the language. Then the con- struction of the chunks which axe a multilin- gual conceptual level allowing the bypass of the limping notion of words. And Finally the discovery of the relations between chunks. We give an overview of the ditferent procedures re- alized and we especially describe the discov- ery of morphemes. This operation is divided into three steps: the discovery of the most fre- quent morphemes of the language. Then the discovery of the other morphemes, and finally the segmentation of the words of the corpus. We concluded with the procedure of correction which required the chunk level. The concepts and algorithms were tested on a twenty nat- ural languages like English, German, Turkish, Vietnamese, Swahili, Finnish, Latin, Indone- sian. 1 Introduction The method presented in this paper is inspired by the distributional approach developed by American structuraIists between 1940 and 1950 (Harris, 1951). This approach is characterized by two facts: (a) the use of corpora and (b) the use of the notion of distri- bution instead of the sense of elements. The distri- bution of an element is the set of the environments in which the element occurs. Other works describe systems that induce structures from corpora, but they use tagged corpora (Brill, 1993), or grammat- ical informations (Brent, 1993), or work with artifi- cial samples (Elman, 1990). Our originality lies in the fazt that we only use untagged and non artificial corpora without specific knowledge about the studied language. We try to discover the structures of a nat- ural language from raw texts of this language (on 100,000 words). We show that this kind of discov- ery is possible if we have some expectations of the structure of Natural Languages and if we use some formal properties. The method relies on structural linguistic con- cepts: the morpheme, the chunk and the linearity of the language, i.e. the corpus is composed of a unidi- mensional sequence of elements. We first give here an overview of the concepts and general principles, from morphemes to syntactic structures discovery. Then we explain in detail how the segmentation is carried out. 2 The General Structure of Sentences Natural Languages are a linear object. It means that sentences are sequences of sounds. In the case of written sentences, we consider them as sequences of letters (or characters). We also consider that lan- guages are not only sequences of sound but are struc- tured in several structural levels. We claim that these different levels are formally indicated in the sentences. How? Since sentences are unidimensional …

Content truncated to 500 words