Urdu Summary Corpus

Urdu Summary Corpus Muhammad Humayoun1, Rao Muhammad Adeel Nawab2, Muhammad Uzair2, Saba Aslam2, Omer Farzand2 1IRIT (Institut de Recherche en Informatique de Toulouse), Universit´e Paul Sabatier, Toulouse, France 2Department of Computer Science, COMSATS Institute of Information Technology, Lahore, Pakistan muhammad.humayoun@irit.fr, adeelnawab@ciitlahore.edu.pk, uzairnaroo@gmail.com, saba 12@hotmail.fr, umerfarzand@gmail.com Abstract Language resources, such as corpora, are important for various natural language processing tasks. Urdu has millions of speakers around the world but it is under-resourced in terms of standard evaluation resources. This paper reports the construction of a benchmark corpus for Urdu summaries (abstracts) to facilitate the development and evaluation of single document summarization systems for Urdu language. In Urdu, space does not always mark word boundary. Therefore, we created two versions of the same corpus. In the first version, words are separated by space. In contrast, proper word boundaries are manually tagged in the second version. We further apply normalization, part-of-speech tagging, morphological analysis, lemmatization, and stemming for the articles and their summaries in both versions. In order to apply these annotations, we re-implemented some NLP tools for Urdu. We provide Urdu Summary Corpus, all these annotations and the needed software tools (as open-source) for researchers to run experiments and to evaluate their work including but not limited to single-document summarization task. Keywords: benchmark corpus, abstracts, automatic text summarization, single-document summarization 1. Introduction Urdu is an Indo-Aryan language, widely spoken in South Asia. It is also spoken all over the world due to the large South Asian Diaspora. Urdu has more than 100 million speakers1. It is written in a modified Perso-Arabic script from right to left. It requires specific rendering to be viewed properly. Normally, it is written in Nastalique, a highly complex writing system that is cursive and context- sensitive. Urdu has a complex morphology that inherits grammatical forms and vocabulary from Arabic, Persian, and native lan- guages of South Asia (Humayoun et al., 2007). There is no capitalization in Urdu. This makes identifying proper nouns, titles, acronyms, and abbreviations a difficult task. Diacritics (vowels) are hardly present in the text and words are guessed with the help of the context of surrounding words. In terms of syntax, it has a free word order (Sub- ject Object Verb). Despite spoken by millions of people, Urdu is an under-resourced language. A sentence illustrat- ing Urdu is given below: � ��� ����� �� ����� �� ���������� ����� Urdu is the national language of Pakistan. The availability of benchmark corpora plays an important role in the development of tools and techniques for var- ious NLP tasks. For automatic text summarization, the shared tasks offered by Document Understanding Confer- ence (DUC)2 and Text Analysis Conference (TAC)3 pro- vide different sets of good quality benchmark corpora mainly for English. These corpora consist of single and 1According to Ethnologue: www.ethnologue.com/ language/urdLast visited: 04-03-2016 2DUC: www-nlpir.nist.gov/projects/duc/ Last visited: 04-03-2016 3TAC: http://www.nist.gov/tac/ Last visited: 04- 03-2016 multi-document summaries written by humans. These sum- maries are abstractive as well as extractive4. These sets of benchmark corpora have been used for …

Content truncated to 500 words