An Unsupervised Method for Discovering Lexical Variations in Roman Urdu Informal Text

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 823–828, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. An Unsupervised Method for Discovering Lexical Variations in Roman Urdu Informal Text Abdul Rafae, ‡Abdul Qayyum, ‡Muhammad Moeenuddin, Asim Karim, †Hassan Sajjad and ‡Faisal Kamiran, †Qatar Computing Research Institute, Hamad Bin Khalifa University Lahore University of Management Sciences, ‡Information Technology University Abstract We present an unsupervised method to find lexical variations in Roman Urdu informal text. Our method includes a phonetic algorithm UrduPhone, a feature- based similarity function, and a clustering algorithm Lex-C. UrduPhone encodes ro- man Urdu strings to their phonetic equiv- alent representations. This produces an initial grouping of different spelling vari- ations of a word. The similarity function incorporates word features and their con- text. Lex-C is a variant of k-medoids clus- tering algorithm that group lexical varia- tions. It incorporates a similarity thresh- old to balance the number of clusters and their maximum similarity. We test our sys- tem on two datasets of SMS and blogs and show an f-measure gain of up to 12% from baseline systems. 1 Introduction Urdu is the national language of Pakistan and one of the official languages of India. It is written in Perso-Arabic script. However in social media and short text messages (SMS), a large proportion of Urdu speakers use roman script (i.e., the English alphabet) for writing, called Roman Urdu. Roman Urdu lacks standard lexicon and usu- ally many spelling variations exist for a given word, e.g., the word zindagi [life] is also written as zindagee, zindagy, zaindagee and zndagi. Specifi- cally, the following normalization issues arise: (1) differently spelled words (see example above), (2) identically spelled words that are lexically differ- ent (e.g., bahar can be used for both [outside] and [spring], and (3) spellings that match words in English (e.g, had [limit] for the English word ‘had’). These inconsistencies cause a problem of data sparsity in basic natural language processing tasks such as Urdu word segmentation (Durrani and Hussain, 2010), part of speech tagging (Saj- jad and Schmid, 2009), spell checking (Naseem and Hussain, 2007), machine translation (Durrani et al., 2010), etc. In this paper, we propose an unsupervised feature-based method that tackles above men- tioned challenges in discovering lexical variations in Roman Urdu. We exploit phonetic and string similarity based features and incorporate contex- tual features via top-k previous and next words’ features. For phonetic information, we develop an encoding scheme for Roman Urdu, UrduPhone, motivated from Soundex. Compared to other available phonetic-based schemes that are mostly limited to English sounds only, UrduPhone maps Roman Urdu homophones effectively. Unlike pre- vious work on short text normalization (see Sec- tion 2), we do not have information about stan- dard word forms in the dataset. The problem be- comes more challenging as every word in the cor- pus is a candidate of every other word. We present a variant of the k-medoids clustering algorithm that forms clusters in which every word has at …

Content truncated to 500 words