ISD at SemEval-2022 Task 6: Sarcasm Detection Using Lightweight Models

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 919 - 922 July 14-15, 2022 ©2022 Association for Computational Linguistics ISD at SemEval-2022 Task 6: Sarcasm Detection Using Lightweight Models Samantha Huang Lynbrook High School Saratoga, CA sam.y.huang2005@gmail.com Ethan A. Chi Stanford University Stanford, CA ethanchi@cs.stanford.edu Nathan A. Chi De Anza College Cupertino, CA chinathan @student.deanza.edu Abstract Robust sarcasm detection is critical for creat- ing artificial systems that can effectively per- form sentiment analysis in written text. In this work, we investigate AI approaches to identi- fying whether a text is sarcastic or not as part of SemEval-2022 Task 6. We focus on creating systems for Task A, where we experiment with lightweight statistical classification approaches trained on both GloVe features and manually- selected features. Additionally, we investigate fine-tuning the transformer model BERT. Our final system for Task A is an Extreme Gradient Boosting Classifier (XGB Classifier) trained on manually-engineered features. Our final sys- tem achieved an F1-score of 0.2403 on Subtask A and was ranked 32 of 43. 1 Introduction Sarcasm is the use of irony–which communicates the opposite of what is said–to humorous and de- risive effect (Bouazizi and Ohtsuki, 2016). On the web, sarcasm is ubiquitous–not least because so- cial media users often apply sarcasm to incorporate a sardonic sense into their statements (Hancock, 2004). This poses a substantial challenge to artifi- cial systems evaluating tasks including sentiment analysis (Liu and Zhang, 2012). It is already a chal- lenge enough for human annotators to determine what is intended to be taken at face-value or not in context-lacking text; it is even more difficult for NLP systems to distinguish between what should be taken literally and what is sarcastic. Task 6 of SemEval-2022 (Abu Farha et al., 2022) provides an environment to build systems to ap- proach these challenges. In particular, Subtask A in Task 6 (Table 1) of SemEval-2022 tests the abil- ity of automated systems to determine whether a text is sarcastic or non-sarcastic. We investi- gate whether lightweight models (which use few computational resources) are able to effectively identify sarcastic speech; we also experiment with fine-tuned Transformer-based models to identify whether larger models perform better. 2 Dataset In Subtask A, we train our models on the official SemEval-2022 Task 6 English training set, which was curated from a set of tweets. Each sentence (examples in Table 6) has been annotated for sar- casm status by the text authors themselves, with 1 denoting a sarcastic text and 0, a non-sarcastic text. Task Description Metric A Determine whether a given F1, text is sarcastic or non-sarcastic. sarcastic-class Table 1: Subtask A overview. 2.1 Train-test Split The dataset has a total of 4868 examples, with 3468 being part of the training set, and 1400 being part of the test set. In total, there are 867 sarcastic and 2601 non-sarcastic texts in the training set. As the testing set labels were not provided until after the competition, we created our own validation set with a 75 : …

Content truncated to 500 words