A Pre-trained Transformer and CNN Model with Joint Language ID and Part-of-Speech Tagging for Code-Mixed Social-Media Text

Proceedings of Recent Advances in Natural Language Processing, pages 367–374 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_042 367 A Pre-trained Transformer and CNN model with Joint Language ID and Part-of-Speech Tagging for Code-Mixed Social-Media Text Suman Dowlagar LTRC IIIT-Hyderabad suman.dowlagar @research.iiit.ac.in Radhika Mamidi LTRC IIIT-Hyderabad radhika.mamidi @iiit.ac.in Abstract Code-mixing (CM) is a frequently observed phenomenon that uses multiple languages in an utterance or sentence. There are no strict grammatical constraints observed in code- mixing, and it consists of non-standard vari- ations of spelling. The linguistic complex- ity resulting from the above factors made the computational analysis of the code-mixed lan- guage a challenging task. Language identi- fication (LI) and part of speech (POS) tag- ging are the fundamental steps that help ana- lyze the structure of the code-mixed text. Of- ten, the LI and POS tagging tasks are inter- dependent in the code-mixing scenario. We project the problem of dealing with multilin- gualism and grammatical structure while ana- lyzing the code-mixed sentence as a joint learn- ing task. In this paper, we jointly train and opti- mize language detection and part of speech tag- ging models in the code-mixed scenario. We used a Transformer with convolutional neural network architecture. We train a joint learn- ing method by combining POS tagging and LI models on code-mixed social media text ob- tained from the ICON shared task. 1 Introduction In bilingual and multilingual communities, code- mixing or code-switching occurs when a person alternates languages below the phrase level inside a sentence or an utterance. Code-mixing(CM) is generally observed in informal settings such as casual conversations or social media text. Code-mixing is defined as mixing phrases, words, and morphemes of one language into an- other language (Myers-Scotton, 1997). Language identification (LI) and part of speech (POS) tagging are the fundamental steps in pro- cessing any code-mixed sentence. LI deals with resolving the language ambiguity of each word in a code-mixed text. POS tagging involves assigning a part of the speech label for each word in a sentence based on its syntactic and semantic information. It helps analyze the grammatical structure of the sentence. Both Language Identification and POS tagging are sequence labeling tasks as they tag each word in a sentence by its corresponding language and POS tags, respectively. While processing a monolingual text, the pri- mary step to understand the sentence’s grammati- cal structure would be POS tagging. However, in the code-mixed scenario, we must consider mul- tilingual phenomena, i.e., each word’s language while POS tagging. Similarly, POS tagging helps capture a better grammatical structure of the text. Moreover, identifying the grammatical structure can improve language identification. In the code- mixing scenario, two tasks go hand in hand. Thus, a joint learning model on POS tagging and LI will considerably enhance the code-mixed text analysis. Recently the transformer models with trans- fer learning such as BERT (Devlin et al., 2018) achieved state-of-the-art accuracy in sequence clas- sification tasks. An evaluation benchmark on code- mixed datasets - GLUECoS (Khanuja et al., 2020) stated that a modified …

Content truncated to 500 words