MirasText: An Automatically Generated Text Corpus for Persian

MirasText: An Automatically Generated Text Corpus for Persian Behnam Sabeti, Hossein Abedi Firouzjaee, Ali Janalizadeh Choobbasti, S.H.E. Mortazavi Najafabadi, Amir Vaheb Miras Technologies International NO. 3, 2nd Alley, North Sheikh Bahai St., Tehran, Iran {behnam, hosein, ali, hani, amir}@miras-tech.com Abstract Natural Language Processing is one of the most important fields of artificial intelligence. The rapid growth of digital content has made this field both practical and challenging at the same time. As opposed to less-resourced languages like Persian, there are several text corpora in dominant languages like English which can be used for NLP applications. In this paper, MirasText which is an automatically generated text corpus for Persian language is presented. In this study, over 250 Persian websites were crawled and several fields like content, description, keywords, title, etc have been extracted to generate MirasText. Topic modeling and language modeling are used to validate the generated corpus. MirasText has over 2.8 million documents and over 1.4 billion tokens, which to our knowledge is the largest Persian corpus currently available. Keywords: Natural Language Processing, Computational Linguistics, Corpus 1 Introduction Natural language processing (NLP) is a field of artificial in- telligence and computational linguistics. It is mainly con- cerned with the interactions between computers and human beings. Many machine learning approaches have been pro- posed to solve NLP problems. In recent years because of the availability of computational resources and the in- troduction of new promising deep learning methods, deep learning approaches have received a lot of attention to solve various NLP problems. Today there are countless online web pages containing textual resources, which can be used to train deep learning models. There are several text corpora available for dominant lan- guages like English. For instance, Google published Google books n-gram corpus (Lin et al., 2012) which con- tains more than 800 billion tokens. Another example is Amazon review dataset (He and McAuley, 2016) which contains more than 142 million reviews crawled from Ama- zon website. Persian language, on the other hand, is one of the less-resourced languages. Although there are some textual datasets for this language but they do not contain huge amount of documents and consequently deep leaning methods that usually require a large corpus of articles to produce valid and reliable results may not be applicable to these corpora. For instance, Hamshahri is a corpus for Per- sian language (AleAhmad et al., 2009) that contains about 300 thousand news articles with semantic tags. Peykare is another Persian corpus (Bijankhan et al., 2011) with more than 100 million words. Although there is enough Persian content on the internet, but the current datasets need to be enriched. In this paper we present MirasText which is an automati- cally generated text corpus for Persian. We crawled more than 250 Persian websites and processed the content to gen- erate this new dataset. MirasText contains contents from web pages with their associated metadata like keywords, description, title, etc. MirasText contains more than 1.4 billion words and can be used for a variety of NLP …

Content truncated to 500 words