Duluth at SemEval-2020 Task 7: Using Surprise as a Key to Unlock Humorous Headlines

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 986–994 Barcelona, Spain (Online), December 12, 2020. 986 Duluth at SemEval-2020 Task 7: Using Surprise as a Key to Unlock Humorous Headlines Shuning Jin1 Yue Yin2 XianE Tang2 Ted Pedersen2 1Rutgers University 2University of Minnesota Duluth shuning.jin@rutgers.edu, {yinxx325,tang0611,tpederse}@d.umn.edu Abstract We use pretrained transformer-based language models in SemEval-2020 Task 7: Assessing the Funniness of Edited News Headlines. Inspired by the incongruity theory of humor, we use a contrastive approach to capture the surprise in the edited headlines. In the official evaluation, our system gets 0.531 RMSE in Subtask 1, 11th among 49 submissions. In Subtask 2, our system gets 0.632 accuracy, 9th among 32 submissions. 1 Introduction Humor detection is a challenging problem in natural language processing. SemEval-2020 Task 7 (Hossain et al., 2020a)1 focuses on detecting humor in English news headlines with micro-edits. Specifically, the edited headlines have one selected word or entity that is replaced by editors, which are then graded by the degree of funniness. Accurate scoring of the funniness from micro-edits can serve as a footstone of humorous text generation (Hossain et al., 2020a). Inspired by the incongruity theory (Veale, 2004; Morreall, 2016), we believe that contrast and surprise is a key ingredient of humor. We instantiate this intuition with a contrastive framework. We then systematically compare three widely used models: CBOW, BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019), providing a benchmark for this task. Our best system, based on RoBERTa, achieves compelling performance for both subtasks. Our code is available on GitHub.2 2 Related Work Early humor recognition systems are mostly based on traditional machine learning methods, such as support vector machine, decision tree, Naive Bayes, and k-nearest neighbors (Castro et al., 2016). Besides, an n-gram language model shows good performance (Yan and Pedersen, 2017) in learning a sense of humor from tweets. Yet n-gram models are limited to a small number of context words. Pretrained language models based on Transformer (Vaswani et al., 2017) can obtain contextual infor- mation of a whole sentence. Among this family, BERT has been used to assess the humor in tweets and jokes (Mao and Liu, 2019; Weller and Seppi, 2019). Enlightened by these recent advances, we use BERT to judge the funniness of edited news headlines. We additionally experiment with RoBERTa, a robustly optimized variant of BERT. Lastly, several works also attempt to explicitly model incongruity and surprise of humourous text, focusing on homophonic puns. Kao et al. (2016) formalizes incongruity as a mixed effect of ambiguity and distinctiveness, quantified by entropy and Kullback-Leibler divergence. He et al. (2019) proposes a local-global surprisal measure based on the log-likelihood ratio, to assess whether a sentence is a pun. However, we focus on a broader definition of humor, and formulate incongruity as an input pair to a dual encoder framework. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1Competition page: https://competitions.codalab.org/competitions/20970 2Code: https://github.com/dora-tang/SemEval-2020-Task-7 987 Task ID Original Headline Edit …

Content truncated to 500 words