Team_Swift at SemEval-2020 Task 9: Tiny Data Specialists through Domain-Specific Pre-training on Code-Mixed Data

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1310–1315 Barcelona, Spain (Online), December 12, 2020. 1310 Team Swift at SemEval-2020 Task 9: Tiny Data Specialists Through Domain-Specific Pre-training On Code-Mixed Data Aditya Malte Department of Computer Engineering, Pune Institute of Computer Technology, Pune, India aditya.malte@gmail.com Pratik Bhavsar TaskHuman Mumbai, India pratik.a.bhavsar@gmail.com Sushant Rathi Department of Computer Science and Engineering Indian Institute of Technology, Delhi, India rathisushant5@gmail.com Abstract Code-mixing is an interesting phenomenon where the speaker switches between two or more languages in the same text. In this paper, we describe an unconventional approach to tackling the SentiMix Hindi-English challenge (uid: aditya_malte). Instead of directly fine-tuning large contemporary Transformer models, we train our own domain-specific embeddings and use them for downstream tasks. We also discuss how this technique provides comparable performance while making for a much more deployable and lightweight model. It should be noted that we have achieved the stated results without using any ensembling techniques, thus respecting a paradigm of efficient and production-ready NLP. All relevant source code shall be made publicly available to encourage the usage and reproduction of the results. 1 Introduction With the rapid growth in the number of active Internet users, there comes a commensurate need to analyze their usage patterns. There may be several reasons to do so, including content moderation, cyber-abuse detection and even for market research purposes to enhance the user experience. However, considering the petabytes of data available online, it becomes a mammoth task that is nearly unachievable through manual means. For instance, social media platforms like Twitter have millions of tweets generated by users every day. By abstracting away tasks like sentiment analysis and content moderation, Natural language processing (NLP) techniques can play an instrumental role in this kind of data analysis. SentiMix Hindi-English (Patwa et al., 2020) is a code-mixed sentiment analysis task. Code-mixing is a phenomenon where the author arbitrarily switches between two or more languages in the same snippet of text. Code-mixing, along with the inherently noisy nature of social media text make its analysis a non- trivial task. We experimented with a wide range of techniques that included contemporary Transformer architectures (Vaswani et al., 2017), word embedding pre-training and language model pretraining. Our experiments provide an interesting inference that near state-of-the-art performance need not require massive model sizes. This analysis would especially be helpful for those trying to deploy machine learning models in production, where computational power and inference speed may be a constraint. 2 Background In this section, we provide a short discussion of the algorithms and technologies that are a part of our implementations. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 1311 2.1 Distributed Word Representations Distributed Word Representations are fixed-sized vectors that abstractly represent the semantics of a word. Word representations rapidly gained popularity and have become an important tool in the NLP toolbox. An example of word embedding includes the word2vec (Mikolov et al., 2013) model that can either be trained …

Content truncated to 500 words