Parsivar: A Language Processing Toolkit for Persian

Parsivar: A Language Processing Toolkit for Persian Salar Mohtaj1, Behnam Roshanfekr2, Atefeh Zafarian3, Habibollah Asghari4 1,2,3,4 ICT Research Institute, Academic Center for Education Culture and Research, Tehran Iran {salar.mohtaj, b.roshanfekr, atefeh.zafarian, habib.asghari}@ictrc.ac.ir Abstract With the growth of Internet usage, a massive amount of textual data is generated on social media and the Web. As the text on the Web are generated by different authors with various types of writing styles and different encodings, a preprocessing step is required before applying any NLP task. The goal of preprocessing is to convert text into a standard format that makes it easy to extract information from documents and sentences. Moreover, the problem is more acute when we deal with Arabic script-based languages, in which there are some different kinds of encoding schemes, different kinds of writing styles and the spaces between or within the words. This paper introduces a preprocessing toolkit named as Parsivar, which is a comprehensive set of tools for Persian text preprocessing tasks. This toolkit performs various kinds of activities comprised of normalization, space correction, tokenization, stemming, parts of speech tagging and shallow parsing. To evaluate the performance of the proposed toolkit, both intrinsic and extrinsic approaches for evaluation have been applied. A Persian plagiarism detection system has been exploited as a downstream task for extrinsic evaluation of the proposed toolkit. The results have revealed that our toolkit outperforms the available Persian preprocessing toolkits by about 8 percent in terms of F1. Keywords: Preprocessing Tasks, Natural Language Processing, Language Processing Toolkit 1. Introduction Preprocessing is one of the essential steps in Natural Language Processing (NLP) tasks that convert unstructured texts into a standard text format suitable for NLP applications. This makes it easy to extract knowledge from documents and sentences. Despite of its importance, few efforts have been accomplished on developing preprocessing tools in low-resource languages. Development of preprocessing tools is more challenging in Arabic script-based languages like Persian, in which there are some difficulties including the lack of enough linguistic resources, various character encodings, spaces between or within multi-token words and a wide variety of suffixes. In this paper, we introduce Parsivar, a Persian preprocessing toolkit that includes a set of tools necessary for different NLP tasks. It normalizes Persian texts to a standard format, corrects the spacing between or within the words, tokenizes words and sentences, extracts word stems, tags the words with their Part of Speech (PoS) and finally performs shallow parsing on sentences. For space correction, it also proposes two different kinds of solutions. The first solution is based on some pre-defined rules and the second one is based on learning methods. The evaluation results have revealed that the proposed toolkit outperforms the other existing preprocessing toolkits in Persian. The rest of the paper is organized as follows: Section 2 discusses the previous works in the field of Persian text preprocessing. In Section 3 we describe the algorithms for implementing each of the modules in Parsivar toolkit in detail. In section 4, we present the experimental …

Content truncated to 500 words