Analyzing Pre-processing Settings for Urdu Single-document Extractive Summarization

Analyzing Preprocessing Settings for Urdu Single-document Extractive Summarization Muhammad Humayoun1, Hwanjo Yu2 1IRIT (Institut de Recherche en Informatique de Toulouse), Universit´e Paul Sabatier, Toulouse, France 2Department of Computer Science & Engineering, Pohang University of Science and Technology (POSTECH), South Korea muhammad.humayoun@irit.fr, hwanjoyu@postech.ac.kr Abstract Preprocessing is a preliminary step in many fields including IR and NLP. The effect of basic preprocessing settings on English for text summarization is well-studied. However, there is no such effort found for the Urdu language (with the best of our knowledge). In this study, we analyze the effect of basic preprocessing settings for single-document text summarization for Urdu, on a benchmark corpus using various experiments. The analysis is performed using the state-of-the-art algorithms for extractive summarization and the effect of stopword removal, lemmatization, and stemming is analyzed. Results showed that these pre-processing settings improve the results. Keywords: automatic text summarization, single-document summarization, extraction based summarization, benchmark experi- ments, preprocessing 1. Introduction Urdu is an Indo-Aryan language, which is widely spoken1. Urdu script is an extended version of Perso-Arabic script. Similar to the other Perso-Arabic scripts, it is written right to left, in a complex, cursive-style writing systems. Urdu inherits a lot of vocabulary from Arabic, Persian and the native languages of South Asia (Humayoun et al., 2007). Due to this influence, Urdu has a complex morphology. In terms of syntax, it has a relatively free word order (Sub- ject Object Verb). Despite spoken by millions of people, Urdu is an under-resourced language in terms of available computational resources. 1.1. Automatic Text Summarization Generally, automatic text summarization addresses the is- sue of generating shortened information from a single doc- ument (or multiple documents written on the same topic). Usually, this shortened text is significantly less than the original text(s) but never more than half of the original text(s) in general (Radev et al., 2002). Two main ap- proaches used for automatic text summarization are ab- straction based and extraction based. Abstraction based approach extracts key points presented in different sen- tences and constructs a coherent summary from the original text by eliminating insignificant details. This requires solv- ing hard questions such as semantic representation, infer- ence, natural language generation, etc. (Radev et al., 2002). In contrast, extraction based technique for summarization is relatively straight forward. It relies on identifying most important sentences from the original document and as- sign them weights (based on their importance). Summary is then formed using top n sentences using these weights. The value of n depends on the length of required sum- mary. These selected sentences are kept intact as units even when some of their parts may not contain most important 1Urdu has more than 100 million speakers, according to Eth- nologue: www.ethnologue.com/language/urd Last vis- ited: 06-03-2016 information. For extraction based technique, several un- supervised methods have been suggested over the years. Some classic examples are (Luhn, 1958) and (Baxendale, 1958; Edmundson, 1969) that use word frequency and sen- tence position respectively. TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and …

Content truncated to 500 words