Optimizing Annotation Effort Using Active Learning Strategies: A Sentiment Analysis Case Study in Persian

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2855–2861 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2855 Optimizing Annotation Effort Using Active Learning Strategies: A Sentiment Analysis Case Study in Persian Seyed Arad Ashrafi Asli, Behnam Sabeti, Zahra Majdabadi, Preni Golazizian, Reza Fahmi and Omid Momenzadeh Miras Technologies International Number 92, Movahed Danesh St., Tehran, Iran {arad, behnam, zahra, preni, reza, omid}@miras-tech.com Abstract Deep learning models are the current State-of-the-art methodologies towards many real-world problems. However, they need a substantial amount of labeled data to be trained appropriately. Acquiring labeled data can be so challenging in some particular domains or less-resourced languages. There are some practical solutions regarding these issues, such as Active Learning and Transfer Learning. Active learning’s idea is simple: let the model choose the samples for annotation instead of labeling the whole dataset. This method leads to a more efficient annotation process. Active Learning models can achieve the baseline performance (the accuracy of the model trained on the whole dataset), with a considerably lower amount of labeled data. Several active learning approaches are tested in this work, and their compatibility with Persian is examined using a brand-new sentiment analysis dataset that is also introduced in this work. MirasOpinion, which to our knowledge is the largest Persian sentiment analysis dataset, is crawled from a Persian e-commerce website and annotated using a crowd-sourcing policy. LDA sampling, which is an efficient Active Learning strategy using Topic Modeling, is proposed in this research. Active Learning Strategies have shown promising results in the Persian language, and LDA sampling showed a competitive performance compared to other approaches. Keywords: Active Learning, Sentiment Analysis, Dataset Generation, Topic Modeling 1. Introduction Despite the notable performance, deep learning approaches are data-hungry. They need huge amounts of data to achieve their optimum performance. This data dependency had become a bottleneck in many real-world applications. Many domains require a specialist to annotate the data sam- ples, for instance, the medical domain. Thus, this annota- tion process can be time-consuming and expensive in many cases. Unfortunately, in less-resourced languages like Per- sian, the above-mentioned problem is even worse, and there are more limitations in attaining related datasets. So there is an increasing need to make the most out of limited avail- able resources. In recent years, some researchers tried to overcome this de- ficiency in the Persian language. For example, MirasText (Sabeti et al., 2018), which is a large Persian corpus with more than 1.4 billion tokens and over 2.8 million docu- ments, is automatically generated. More specifically, in the sentiment analysis domain, there were some efforts in en- riching sentiment lexicon for Persian, such as LexiPers (Sa- beti et al., 2019), PerSent (Dashtipour et al., 2020), Senti- Fars (Dehkharghani, 2019), and HesNegar (Asgarian et al., 2018). However, these lexicons are mainly used in unsu- pervised sentiment analysis methods. Despite those developments, insufficient resources remain as the main difficulty in many domains in Persian. Active learning is one of …

Content truncated to 500 words