Kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 817–823 Barcelona, Spain (Online), December 12, 2020. 817 kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification Jiaxiang Liu, Xuyi Chen, Shikun Feng, Shuohuan Wang, Xuan Ouyang, Yu Sun, Zhengjie Huang, Weiyue Su Baidu Inc., China {liujiaxiang, chenxuyi, fengshikun01, wangshuohuan}@baidu.com {ouyangxuan, sunyu02, huangzhengjie, suweiyue}@baidu.com Abstract Code switching is a linguistic phenomenon that may occur within a multilingual setting where speakers share more than one language. With the increasing communication between groups with different languages, this phenomenon is more and more popular. However, there are little research and data in this area, especially in code-mixing sentiment classification. In this work, the domain transfer learning from state-of-the-art uni-language model ERNIE is tested on the code-mixing dataset, and surprisingly, a strong baseline is achieved. Furthermore, the adversarial training with a multi-lingual model is used to achieve 1st place of SemEval-2020 Task 9 Hindi-English sentiment classification competition. 1 Introduction In today’s society, the use of social media has become a necessary daily activity. As a result, a large amount of text content has been produced. Among those corpora, there are many code-mixed texts. Code-mixing is a common phenomenon in societies in which two or more languages are used. It is the change of one language to another within the same utterance or in the same oral/written text (Ho and others, 2007). Code-mixing poses several unseen difficulties in NLP tasks on lexical/syntax/semantic levels. In recent years, a lot of models based on Transformer (Vaswani et al., 2017) have been proposed such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and ALBERT (Lan et al., 2019). Those models take advantage of both available large corpora and computational power and have outperformed the traditional techniques in various sentiment analysis tasks (Kim, 2014; Hochreiter and Schmidhuber, 1997). Meanwhile, many multi-lingual models are proposed, e.g. (Ma et al., 1999; Conneau et al., 2019). (Sarkar et al., 2019) have tested the multilingual BERT with Hierarchical Attentive Network, Shakeel and Karim (2020) have tested BERT for code-mixing informal short text classification. However, the code-mixed corpus is small. Large models like BERT (Devlin et al., 2019) and ERNIE (Sun et al., 2019) have millions of parameters, and are prone to overfit. To tackle this problem, adversarial training can be used as a regularizer in the training phase (Szegedy et al., 2013; Goodfellow et al., 2014). Adversarial examples are a way of fooling a neural network to behave incorrectly (Szegedy et al., 2013). They are created by applying a small perturbation to the original inputs. The original algorithm was designed for use in the image field, and the perturbation added to the image was deliberately designed to be invisible to the naked eye, but causing neural networks to output a completely different response from the true one. By introducing those adversarial examples, which the neural nets make mistakes on, to the network during training, the performance can be improved. This is called “adversarial training” which acts as a …

Content truncated to 500 words