Amobee at SemEval-2020 Task 7: Regularization of Language Model Based Classifiers

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 981–985 Barcelona, Spain (Online), December 12, 2020. 981 Amobee at SemEval-2020 Task 7: Regularization of Language Model Based Classifiers Alon Rozental Dadi Biton Amobee Inc., Tel Aviv, Israel alon.rozental,dadi.biton,ido.blank@amobee.com Ido Blank Abstract This paper describes Amobee’s participation in SemEval-2020 task 7: “Assessing Humor in Edited News Headlines”, sub-tasks 1 and 2. The goal of this task was to estimate the funniness of human modified news headlines. In this paper we present methods to fine-tune and ensemble various language models (LM) based classifiers for this task. This technique used for both sub-tasks and reached the second place (out of 49) in sub-tasks 1 with RMSE score of 0.5, and the second (out of 32) place in sub-task 2 with accuracy of 66% without using any additional data except the official training set. 1 Introduction While most of the studies are interested in whether a chunk of text is funny, this study examined whether an atomic change can transform a non-funny headline into a funny one. The study focused on two sub-tasks. The first task is to the determine how funny the micro change in the headline is. The second task is to choose between two micro changes of the original headline, which one is funnier. This task is part of SemEval 2020 workshop (Hossain et al., 2020). There were 49 groups in the first sub-task and 32 groups in the second. Contextual word embeddings is currently one of the most popular ways to create a numeric representa- tion of a document. It is capable of capturing the context of a word in a document, semantic and syntactic similarity, relation with other words, etc. Pre-trained models of this kind, such as (Liu et al., 2019; Yang et al., 2019; Devlin et al., 2018), can be fine-tuned to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. However, due to the fact that these models usually have a large number of parameters, over 300 million for the models mentioned above, they tend to overfit very quickly when used without any modification on small datasets. For example, BERT-large (Devlin et al., 2018) has over 30,000 times more parameters than the SemEval-2020-task-7-dataset has training examples. In this paper, we present a way to improve the performance of systems that are based upon language models (LM) or contextual token representation models by adding an L2-regularization term on the difference between the pre-trained LM and the fine-tuned model. For convenience, we’ll use the term LM in this paper despite the fact some of the models we refer to are word representation models and not proper language models. The paper is organized as follows: Section 2 describes the background for this task and an explanation of the dataset. Section 3 describes our system architecture and presents the L2-regularization loss for training the language model and our ensemble method. Section 4 describes our experimental setup, and Section …

Content truncated to 500 words