LSCP: Enhanced Large Scale Colloquial Persian Language Understanding

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6323–6327 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6323 LSCP: Enhanced Large Scale Colloquial Persian Language Understanding Hadi Abdi Khojasteh1, Ebrahim Ansari 1,2, Mahdi Bohlouli 1,3 1 Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences (IASBS), Zanjan, Iran 2 Institute of Formal and Applied Linguistics ( ´UFAL), Faculty of Mathematics and Physics, Charles University in Prague, Czechia 3 Research and Innovation Department, Petanux GmbH, Germany {hkhojasteh, ansari, bohlouli}@iasbs.ac.ir Abstract Language recognition has been significantly advanced in recent years by means of modern machine learning methods such as deep learning and benchmarks with rich annotations. However, research is still limited in low-resource formal languages. This consists of a significant gap in describing the colloquial language especially for low-resourced ones such as Persian. In order to target this gap for low resource languages, we propose a “Large Scale Colloquial Persian Dataset” (LSCP). LSCP is hierarchically organized in a semantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. This encompasses the recognition of multiple semantic aspects in the human-level sentences, which naturally captures from the real-world sentences. We believe that further investigations and processing, as well as the application of novel algorithms and methods, can strengthen enriching computerized understanding and processing of low resource languages. The proposed corpus consists of 120M sentences resulted from 27M tweets annotated with parsing tree, part-of-speech tags, sentiment polarity and translation in five different languages. Keywords: Persian Corpus, Large Scale Language Understanding, Colloquial Persian Language, Multilingual, Informal Language 1. Introduction Language understanding is a comprehensive problem that encompasses the recognition of multiple semantic aspects of relations between different linguistic units and com- pounds: homonymy, synonymy, antonymy, hypernymy, hy- ponymy, meronymy, metonymy, holonymy, paronyms. In other hands, large scale corpora are one of the key resources in Natural Language Processing (NLP). In spite of their im- portance in many lingual applications, no large-scale Per- sian corpus has been made available so far, given the diffi- culties in its creation and the intensive labour required due to its low resources, especially in colloquialism. Today, modern machine learning algorithms such as deep neural networks provide improved processing performance for a wide variety of applications, mainly text mining and NLP. But, they need large scale datasets, specially for train- ing and pre-processing of the algorithms. It should also be mentioned that collecting, preparing, pre-processing and annotating large scale datasets from social media and web based materials such as product reviews are not easy task. Because most of them are associated with the mixed mul- tilingual and informal texts that need better understanding. Social networks such as Twitter deliver large volume of dis- parate data including short and long mixed and informal text consisting symbols, other languages and images. The quality of machine translation depends heavily on availability of various datasets. In this regard, low re- sourced languages such as Persian lack the …

Content truncated to 500 words