Urdu Word Segmentation

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 528–536, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Urdu Word Segmentation Nadir Durrani Sarmad Hussain Institute for NLP Center for Research in Urdu Language Processing Universität Stuttgart National University of Computer and Emerging Sciences durrani@ims.uni-stuttgart.de sarmad.hussain@nu.edu.pk Abstract Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that face word segmenta- tion challenge. However, unlike other Asian languages, word segmentation in Urdu not only has space omission errors but also space insertion errors. This paper dis- cusses how orthographic and linguistic features in Urdu trigger these two problems. It also discusses the work that has been done to tokenize input text. We employ a hybrid solution that performs an n-gram ranking on top of rule based maximum matching heuristic. Our best technique gives an error detection of 85.8% and over- all accuracy of 95.8%. Further issues and possible fu- ture directions are also discussed. 1 Introduction All language processing applications require input text to be tokenized into words for further processing. Languages like English normally use white spaces or punctuation marks to identify word boundaries, though with some complications, e.g. the word “e.g.” uses a period in between and thus the period does not indicate a word boundary. However, many Asian languages like Thai, Khmer, Lao and Dzongkha do not have word boundaries and thus do not use white space to consistently mark word endings. This makes the process of tokenization of input into words for such languages very challenging. Urdu is spoken by more than 100 million people, mostly in Pakistan and India1. It is an Indo-Aryan language, written using Arabic script from right to left, and Nastalique writing style (Hussain, 2003). 1 Ethnologue.com http://www.ethnologue.com/14/show_language.asp?code=UR D Nastalique is a cursive writing system, which also does not have a concept of space. Thus, though space is used in typing the language, it serves other purposes, as discussed later in this paper. This en- tails that space cannot be used as a reliable delimi- ter for words. Therefore, Urdu shares the word segmentation challenge for language processing, like other Asian languages. This paper explains the problem of word segmen- tation in Urdu. It gives details of work done to investigate linguistic typology of words and moti- vation of using space in Urdu. The paper then presents an algorithm developed to automatically process the input to produce consistent word seg- mentation, and finally discusses the results and future directions. 2 Urdu Writing System Urdu is written in cursive Arabic script. Characters in general join with the neighbors within a word and in doing so acquire different shapes. Logically, a character can acquire up to four shapes, i.e. ini- tial, medial, final position in a connected sequence or an isolated form. The characters having this four-way shaping are known as joiners. However, another set of …

Content truncated to 500 words