HIT - A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4625–4639 August 1–6, 2021. ©2021 Association for Computational Linguistics 4625 HIT: A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation Ayan Sengupta1, Sourabh Kumar Bhattacharjee1, Tanmoy Chakraborty2, Md Shad Akhtar2 1Optum Global Advantage (UnitedHealth Group), Noida, India 2Dept. of CSE, IIIT-Delhi, India {ayan.sengupta007, sourabhb398}@gmail.com; {tanmoy, shad.akhtar}@iiitd.ac.in Abstract Understanding linguistics and morphology of resource-scarce code-mixed texts remains a key challenge in text processing. Although word embedding comes in handy to support downstream tasks for low-resource languages, there are plenty of scopes in improving the quality of language representation particularly for code-mixed languages. In this paper, we propose HIT, a robust representation learn- ing method for code-mixed texts. HIT is a hierarchical transformer-based framework that captures the semantic relationship among words and hierarchically learns the sentence- level semantics using a fused attention mech- anism. HIT incorporates two attention mod- ules, a multi-headed self-attention and an outer product attention module, and computes their weighted sum to obtain the attention weights. Our evaluation of HIT on one European (Span- ish) and five Indic (Hindi, Bengali, Tamil, Tel- ugu, and Malayalam) languages across four NLP tasks on eleven datasets suggests signif- icant performance improvement against var- ious state-of-the-art systems. We further show the adaptability of learned representation across tasks in a transfer learning setup (with and without fine-tuning). 1 Introduction India is the second most populated country in the world, where ∼ 1.36 billion people speak in over 200 different languages. Among them, the top five languages (Hindi, Bengali, Telegu, Tamil, and Malayalam) covers ∼ 93%of the entire population with more than 26% of them being bilingual (as per Wikipedia). Moreover, a significant propor- tion of them (Singh et al., 2018a) use code-mixed languages to express themselves in Online Social Networks (OSN). Code-mixing (CM) is a linguistic phenomenon in which two or more languages are alternately used during conversation. One of the languages is usually English, while the other can be any re- gional language such as Hindi (Hindi + English → Hinglish), Bengali (Bengali + English → Benglish), Spanish (Spanish + English → Spaniglish), etc. Their presence on social media platforms and in day-to-day conversions among the people of a multi-lingual communities (such as Indians) is over- whelming. Despite the fact that a significant popu- lation is comfortable with code-mixed languages, the research involving them is fairly young. One of the prime reasons is the linguistic diversity, i.e., re- search on any language often fails to adapt for other distant languages, thus they need to be studied and researched separately. In recent years, many orga- nizations have identified the challenges and have put in commendable efforts for the development of computational systems in regional monolingual or code-mixed setups. Traditionally, the NLP community has studied the code-mixing phenomenon from a task-specific point of view. Recently, a few studies (Pratapa et al., 2018; Aguilar and Solorio, 2020) have started learning representations for code-mixed texts for semantic and syntactic tasks. While the former has showcased …

Content truncated to 500 words