Sam Miller at SemEval-2023 Task 5: Classification and Type-specific Spoiler Extraction Using XLNET and Other Transformer Models

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1217–1224 July 13-14, 2023 ©2023 Association for Computational Linguistics Sam Miller at SemEval-2023 Task 5: Classification and Type-Specific Spoiler Extraction using XLNET and other Transformer Models Pia Bernards TH Köln pia.stoermer@smail.th-koeln.de Tobias Esser TH Köln tobias_vincent.esser@smail.th-koeln.de Patrick Thomasius TH Köln patrick_christian.thomasius@smail.th-koeln.de Abstract This paper proposes an approach to classify and generate spoilers for clickbait articles and posts. For the spoiler classification, XLNET was trained to fine-tune a model. With an accu- racy of 0.66, 2 out of 3 spoilers are predicted accurately. The spoiler generation approach in- volves preprocessing the clickbait text and post- processing the output to fit the spoiler type. The results were evaluated on a test dataset of 1000 posts, with the best result for spoiler genera- tion achieved by fine-tuning a RoBERTa Large model with a small learning rate and sample size, reaching a BLEU score of 0.311. The paper provides an overview of the models and techniques used and discusses the experimental setup. 1 Introduction In advertising and social media, there are common patterns and tactics that are employed to capture and capitalize the attention of consumers. One of those is the use of so-called “Clickbait”, which cre- ate curiosity in the users that motivate them to click on the articles (Hagen et al., 2022a), (Loewenstein, 1994). The proposed solution, therefore, is to gener- ate an approach that automates the generation of a spoiler, that spoils the information needed to close the users curiosity gap. The user can then make a more informed decision whether they want to pro- ceed. The clickbait posts and articles are mostly in English language. The task is split into two parts, in the first task, the type of the required spoiler is determined, whether it’s a shorter phrase, a longer passage or a multipart spoiler. In the second part, the actual spoilers will be generated or extracted. Given task 2, our approach was to fine-tune a question answering model and use post-processing the respective spoiler type, so that the format and length of the answer fits the requested spoiler type. For multipart spoilers, recursive spoiler generation was used to generate multiple answers. Further- more, the possibility of doing preprocessing in the form of query rewriting was explored. We tried different pretrained models, and larger models with more pretraining (RoBERTa Large Squad-2) outperformed smaller models signifi- cantly. Our result for task 2 was comparable to the other groups, but worse than the Baseline (0.311 BLEU on our approach compared to 0.38 on the transformer baseline (Hagen et al., 2022b)). Models were added to the Docker Images (Merkel, 2014) to ensure replicability. 1.1 Data The training data consists of 3200 posts, the valida- tion dataset contains 800 posts and the test dataset, which was private during development, contains another 1000 posts. Most spoilers are extracted, with a small minority of abstractive spoilers in train and evaluation datasets. Multipart spoilers are supposed to consist of 5 items that can stem …

Content truncated to 500 words