Developing Punjabi Morphology, Corpus and Lexicon

Developing Punjabi Morphology, Corpus and Lexicon Muhammad Humayoun Laboratory of Mathematics University of Savoie, France. mhuma@univ-savoie.fr Aarne Ranta Department of CS and Engineering Chalmers University of Technology and University of Gothenburg, Sweden. aarne@chalmers.se Abstract. We describe an implementation of morphology, development of a corpus and building of a lexicon for Punjabi language. Such resources are building blocks for various language technology tasks ranging from part of speech tagging to machine translation. Their importance is further increased by the fact that Punjabi is an under resourced language. We release these resources as open-source. Keywords: Morphology, Corpus building, Lexicon extraction, Punjabi, Shahmukhi. 1 Introduction Punjabi is an Indo-Aryan language, widely spoken in the Punjab region of south Asia. It is spoken by 88 million people (Lewis 2009), which makes it approximately the 13th most widely spoken language in the world. Furthermore, it is the most widely spoken language of Pakistan as regards the number of native speakers. Punjabi is normally written in two scripts: Gurmukhi (in India) and Shahmukhi (in Pakistan). The resources reported in this paper are developed using Shahmukhi. Shahmukhi is a variant of the Perso- Arabic scripts. Therefore it possesses most of their inherent characteristics such as right to left writing, an optional use of diacritic marks and short vowels being not considered as letters of their own but applied above or below a consonant by using appropriate diacritics. It has 49 consonants, 16 diacritical marks and 16 vowels; see Malik (2006) for a detailed account of both scripts. In this paper, we report three important resources which are building blocks for various language technology tasks ranging from part of speech tagging to machine translation. First, we report an implementation of inflectional morphology for Punjabi which is described in Section 3. We use GF (Grammatical Framework, Ranta 2004) which is a framework for developing multilingual grammar applications. GF also provides a built-in morphological analyzer and generator for the implemented languages. An overview of GF is given in Section 2. Second, we report the development of a corpus containing 0.9 million words (941,284), which is collected partly from Wikipedia, as described in section 4. Finally, we report a lexicon of 13,600 words (named entities:63%, lemmas of inflected words:37%; a lemma is also known as a dictionary form or a base form.). It is automatically extracted from the corpus and manually cleaned, as reported in Section 4. We have used a method proposed by (Forsberg et al., 2006) for this extraction. We describe our procedure in section 5. We release these resources as open-source1. Their importance is further increased by the fact that Punjabi is an under resourced language. For these components, our source of study is the Mahji dialect of Punjabi which is spoken in and around Lahore and is considered to be the standard form (Akhtar 1999:10). We used simple Roman2 transcription for Punjabi. 1 Downloadable from homepage: www.lama.univ-savoie.fr/~humayoun/punjabi 2 ‘a’, ‘i’, ‘o’ are used for short vowels; ‘aa’, ‘ii’ and ‘oo’ are used for the long ones. Capital …

Content truncated to 500 words