WUY at SemEval-2020 Task 7: Combining BERT and Naive Bayes-SVM for Humor Assessment in Edited News Headlines

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1071–1076 Barcelona, Spain (Online), December 12, 2020. 1071 WUY at SemEval-2020 Task 7: Combining BERT and Naive Bayes-SVM for Humor Assessment in Edited News Headlines Cheng Zhang Graduate School of Fundamental Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo, 169-8555 Japan {zchelllo, yamana}@yama.info.waseda.ac.jp Hayato Yamana Faculty of Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo, 169-8555 Japan Abstract This paper describes our participation in SemEval 2020 Task 7 on assessment of humor in edited news headlines, which includes two subtasks, estimating the humor of micro-editd news headlines (subtask A) and predicting the more humorous of the two edited headlines (subtask B). To address these tasks, we propose two systems. The first system adopts a regression-based fine-tuned single-sequence bidirectional encoder representations from transformers (BERT) model with easy data augmentation (EDA), called “BERT+EDA”. The second system adopts a hybrid of a regression-based fine-tuned sequence-pair BERT model and a combined Naive Bayes and support vector machine (SVM) model estimated on term frequency–inverse document frequency (TFIDF) features, called “BERT+NB-SVM”. In this case, no additional training datasets were used, and the BERT+NB-SVM model outperformed BERT+EDA. The official root-mean-square deviation (RMSE) score for subtask A is 0.57369 and ranks 31st out of 48, whereas the best RMSE of BERT+NB-SVM is 0.52429, ranking 7th. For subtask B, we simply use a sequence-pair BERT model, the official accuracy of which is 0.53196 and ranks 25th out of 32. 1 Introduction Humor, a high-level form of human communication, is omnipresent, including in the social media as well as real-life situations. The proper use of humor can have a positive impact on our lives. An automatic identification of humor can help in better understanding the structure and theory of humor. Moreover, correctly understanding humor is important for improving the performance of many natural language processing applications such as a sentiment analysis and intention mining. To evaluate the progress of automatic humor assessment, SemEval-2020 task 71 aims to study how machines can understand humor generated by applying short edits to news headlines (Hossain et al., 2020). A dataset, called Humicroedit (Hossain et al., 2019), with a total of 15,095 English edited news headlines collected from Reddit (reddit.com) along with mean humor scores, was developed. With this dataset, humor is generated after a short editing, e.g., “President vows to cut taxes hair”. Subtask A focuses on the prediction of the mean humor of an edited headline, and subtask B involves predicting the funnier of two edited headlines. To address this problem, we propose two systems2. The first (BERT+EDA) is a regression fine-tuned single sequence bidirectional encoder representations from transformers (BERT) model (Devlin et al., 2019) with easy data augmentation (EDA) tool3, which is an implementation of the data augmentation theory of Wei and Zou (2019). The second (BERT+NB-SVM) is a hybrid of the regression fine-tuned sequence-pair BERT and a Naive Bayes-Support Vector Machine (NB-SVM) model (Wang and Manning, 2012) estimated on TFIDF features. Here we combine the strength of deep …

Content truncated to 500 words