Classification of Code-Mixed Text Using Capsule Networks

Proceedings of Recent Advances in Natural Language Processing, pages 256–263 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_030 256 Classification of Code-Mixed Text Using Capsule Networks Shanaka Chaturanga, Surangika Ranathunga Department of Computer Science and Engineering, University of Moratuwa, Katubedda 10400, Sri Lanka. [shanaka.chathuranga.19, surangika]@cse.mrt.ac.lk Abstract A major challenge in analysing social me- dia data belonging to languages that use non- English script is its code-mixed nature. Recent research has presented state-of-the-art contex- tual embedding models (both monolingual s.a. BERT and multilingual s.a. XLM-R) as a promising approach. In this paper, we show that the performance of such embedding mod- els depends on multiple factors, such as the level of code-mixing in the dataset, and the size of the training dataset. We empirically show that a newly introduced Capsule+biGRU classifier could outperform a classifier built on the English-BERT as well as XLM-R just with a training dataset of about 6500 samples for the Sinhala-English code-mixed data. 1 Introduction Social media has become very popular among peo- ple across the world during the last decade, mainly due to the popularity of smart mobile phones. For low-resource languages, this social media data has become a major source of text in building Natural Language Processing (NLP) applications. Most of the content in social media tends to be informal. When the users are (at least to a certain degree) multilingual, the informal content they publish in social media tends to be code-mixed. Code-mixed data is a result of code-switching, which denotes a shift from one language to another within a single utterance (Sitaram et al., 2019). Recent work on text classification with code-mixed data has used contextual embedding models such as BERT (Devlin et al., 2019a), and their multilin- gual versions, such as mBERT or XLM-R (Aguilar et al., 2020; Kumar et al., 2020). However, using these pre-trained models for code-mixed text clas- sification, in particular the domain-specific data in low-resource languages poses many challenges. Contextual embedding models such as BERT need large volumes of monolingual data to train. On the other hand, not every language is included in the pre-trained multilingual models, and low-resource languages are underrepresented in those models (due to the smaller amounts of low-resource lan- guage data used in contrast to high-resource lan- guages when training these models). In fact, there is a line of research that has shown even simple classifiers such as Logistic Regression proving to be more effective than the multilingual embedding models (Chakravarthi et al., 2020). In this paper, we empirically show that the success of contextual embedding models on code-mixed text classification depends on multiple factors, and they can indeed be sub-optimal compared to text classification based on neural models other than transformer based ones. We selected Sinhala and Malayalam, which are low-resource languages. In the recent language categorization by Joshi et al. (2020), Sinhala be- longs to class 0 (i.e. it has exceptionally limited resources). Malayalam is categorised as 1, (i.e. it has some unlabelled data, however collecting la- belled data is challenging). A food recipe dataset with …

Content truncated to 500 words