Multilingual document language recognition for creating corpora

MT Summit VII Sept. 1999 Multilingual Document Language Recognition for Creating Corpora Yevgeny Ludovik Ron Zacharski New Mexico State University USA Abstract In this paper we describe a language recogni- tion algorithm for multilingual documents that is based on mixed-order n-grams, Markov chains, maximum likelihood, and dynamic programming. We present the re- sults of an experimental study that showed that the performance of this algorithm has practical value. 1 Introduction Language recognition algorithms are an essential component of many natural language processing sys- tems. For example, a system that translates web- documents must first determine the language of that document. Moreover, in multilingual documents, the system must determine monolingual segments and the language of each segment. Language recognition al- gorithms are also essential in the development of many natural language processing systems. For exam- ple, very large corpora are often required in the de- velopment of machine translation systems and a lan- guage identification system can be combined with a web spider to automatically collect such corpora. These recognition algorithms are not just concerned with the language of electronic texts, but also with how the characters of that language are encoded. This specification is important because the characters of a given language may be represented by several distinct encoding methods or code sets. For example, Japanese text may be in J1S-X-0208-1990 or JIS-X-0212-1990, among others, and Chinese text might be in GB-2312- 80 or Big Five. Thus, throughout this paper we will always be concerned with identifying a specific lan- guage/code set pair. However, to ease exposition we use the term language to mean this language/code set pair. In the following sections we describe a language recognition algorithm for multilingual documents based on mixed order n-grams, Markov chains, maximum likelihood, and dynamic programming. We present the results of a study that examined the per- formance of this algorithm on multilingual docu- ments, which consisted of fragments in 30 languages. We discuss how the algorithm can be incorporated into a web spider that collects multilingual documents in specified languages. 2 Recognition Algorithm for Mono- lingual Documents A common sense and frequently used approach to language identification has been to use lists of words that commonly appear in the languages of interest (Ingle 76).That is, for every language of interest one constructs a list containing the most frequent words in that language. For example, the, of, and and are com- mon words in English, and el, de, que, and y are common words in Spanish. If you are trying to iden- tify the language of a text and you see a lot of the's of's and and's in the text a good guess would be that the text is in English. Similarly, if you see el, que, and de, a good guess would be Spanish. While this ap- proach is simple, intuitive, and performs adequately in many cases, it does have several drawbacks. First, while the approach works well for identifying texts of moderate length, its performance significantly de- grades as the …

Content truncated to 500 words