Ambiguous (((Par(t)(it))((ion))(s))(in)) Thai Text

Language, Information and Computation(PACLIC 11)1996, 109-118 Ambiguous (((Par(t)(it))((ion))(s))(in)) Thai Text Doug Cooper Center for Research in Computational Linguistics, Bangkok doug@nwg.nectec.or.th http://seasrc.th.net Abstract Despite the importance of segmentation to a variety of software applications, almost noth- ing is known about the characteristics or distribution of ambiguous partitions (eg. to Pend vs. top_end) in Thai text. By using special-purpose code to investigate a large (-400K word) text corpus, we were able to extract 36,267 such sequences, involving 9,253 distinct examples. Of these, a little more than two-fifths involved genuinely ambiguous partitions. We classify partitioning problems into distinct categories, report on many of their statisti- cal and lexical characteristics, and describe heuristics for choosing the correct partition that do not depend on the availability of a large segmented corpus. 1. Introduction Thai writing does not use spaces to segment text into words. While open text contains many obvious separation points (bigdog vs. big dog), and a smaller group of question- able bind points that are usually permissible either way (toolbox vs. toolbox), there is inevitably a residue of ambiguous partition points (to_pend vs. top_end) for which computer segmentation is essentially random. This causes difficulty for many software applications: line-breaking, spell-checking, machine-assisted translation, text-to-speech, optical character recognition, full-text indexing, corpus-based dictionaries, etc. Yet despite the importance of segmentation, little is known about the characteristics of ambiguous partitions in Thai, or its orthographic cousins Khmer, Lao, and Burmese. Work has been slow and progress poor due to a lack of formal, concrete analysis. We know the problem's gross characteristics, and the general direction of solutions, but there are few theories to guide the way or allow comparison of research results. This paper describes experiments carried out on a 2 megabyte (roughly 400,000 word) Thai corpus. We collected nearly ten thousand distinct alternative segmentations of at least two words in length. Of these, a little more than two-fifths involved genuinely ambiguous partitions. We classify partitioning problems into distinct categories, report on statistical and lexical characteristics of ambiguous partitions, and describe heuristics for disambiguating that do not depend on the availability of a large segmented corpus. Our results make several contributions to understanding Thai text segmentation. First, we categorize breakpoints in a way that distinguishes between choices that are and are not semantically significant, and show how to collect them automatically and consis- tently. Second, we find that ambiguous instances are fairly rare (roughly 5% of word break opportunities), and have a pronounced Zipfian distribution — a relatively small number of circumstances produce a great deal of ambiguity; and show ways of collect- ing low-frequency items that exhibit the same behavior. Third, we find that contrary to the canonical examples, resolving ambiguity does not usually depend on knowing or un- derstanding the context it occurs in. Finally, we suggest new methods — stop nodes, go collocates, and analysis of hidden 'swing strings' — to aid in automated disambiguation. 2. Prior Work There is extensive literature on text segmentation for Asian languages; those for Chinese are typical (Wu …

Content truncated to 500 words