CS/NLP at SemEval-2022 Task 4: Effective Data Augmentation Methods for Patronizing Language Detection and Multi-label Classification with RoBERTa and GPT3

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 503 - 508 July 14-15, 2022 ©2022 Association for Computational Linguistics CS/NLP at SemEval-2022 Task 4: Effective Data Augmentation Methods for Patronizing Language Detection and Multi-label Classification with RoBERTa and GPT3 Daniel Saeedi University of Tehran saeedi.danial@ut.ac.ir Sirwe Saeedi Western Michigan University sirwe.saeedi@wmich.edu Aliakbar Panahi C3 AI ali.panahi@c3.ai Alvis C.M. Fong Western Michigan University alvis.fong@wmich.edu Abstract This paper presents a combination of data aug- mentation methods to boost the performance of state-of-the-art transformer-based language models for Patronizing and Condescending Language (PCL) detection and multi-label PCL classification tasks. These tasks are inherently different from sentiment analysis because posi- tive/negative hidden attitudes in the context will not necessarily be considered positive/negative for PCL tasks. The oblation study observes that the imbalance degree of PCL dataset is in the extreme range. This paper presents a modified version of sentence paraphrasing deep learning model (PEGASUS) to tackle the limitation of maximum sequence length. The proposed algo- rithm has no specific maximum input length to paraphrase sequences. Our augmented under- represented class of annotated data achieved competitive results among top-16 SemEval- 2022 participants. This paper’s approaches rely on fine-tuning pretrained RoBERTa and GPT3 models such as Davinci and Curie engines with extra-enriched PCL dataset. Furthermore, we discuss Few-Shot learning technique to over- come the limitation of low-resource NLP prob- lems.1 Keywords: Natural Language Processing, Transformers, Data Augmentation, RoBERTa, GPT-3, Curie and Davinci Engines. 1 Introduction Natural Language Understanding (NLU) and Inter- pretation (NLI) is a branch of Natural Language Processing (NLP) in Artificial Intelligence (AI), which involves understanding and analyzing human language in-depth. Recent advances in Deep Neu- ral Networks (DNNs) have enabled NLP research scientists to achieve state-of-the-art results for tasks 1Our implementation is publicly available at https://github.com/daniel-saeedi/PCL_ Detection_SemEval2022 that were extremely difficult, if not impossible De- vlin et al. (2019), Lan et al. (2020). However, un- derstanding human emotions, reactions, and uncov- ering hidden insights from unstructured text data such as news stories channel is still challenging. Language attitudes and intentions extracting in response to the support for the marginalized and vulnerable communities is one of the emergent NLP applications. Patronizing and condescending language (PCL) is a type of behavior that projects a sense of superiority to vulnerable populations Pérez-Almendros et al. (2020). Furthermore, biases and discrimination can result from patronizing atti- tudes, causing some people to feel unfairly treated, inadequate, unintelligent, and possibly infuriated Saeedi et al. (2021). Since raw text data extracting from web is a common data collection method, language mod- els can learn different forms of harmful language Heidari and Jones (2020). The PCL understand- ing is inherently different from sentiment analysis because positive/negative hidden attitudes in the context will not necessarily be considered posi- tive/negative for PCL tasks. It is difficult due to the fair amount of world knowledge and commonsense reasoning required to understand this kind of lan- guage Saeedi et al. (2020). The fine-grained idea of PCL detection towards vulnerable communities was presented by Pérez-Almendros …

Content truncated to 500 words