SemEval-2020 Task 7: Assessing Humor in Edited News Headlines

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 746–758 Barcelona, Spain (Online), December 12, 2020. 746 SemEval-2020 Task 7: Assessing Humor in Edited News Headlines Nabil Hossain†, John Krumm‡, Michael Gamon‡ and Henry Kautz† †Department of Computer Science, University of Rochester ‡Microsoft Research AI, Microsoft Corporation, Redmond, WA {nhossain,kautz}@cs.rochester.edu, {jckrumm,mgamon}@microsoft.com Abstract This paper describes the SemEval-2020 shared task “Assessing Humor in Edited News Headlines.” The task’s dataset contains news headlines in which short edits were applied to make them funny, and the funniness of these edited headlines was rated using crowdsourcing. This task includes two subtasks, the first of which is to estimate the funniness of headlines on a humor scale in the interval 0-3. The second subtask is to predict, for a pair of edited versions of the same original headline, which is the funnier version. To date, this task is the most popular shared computational humor task, attracting 48 teams for the first subtask and 31 teams for the second. 1 Introduction (a) The Headline Editing Interface. (b) The Headline Rating Interface. Figure 1: The funny headline data annotation interfaces. When editing, only the underlined tokens are replaceable. Humor is an important ingredient of human communication, and every automatic system aiming at emulating human intelligence will eventually have to develop capabilities to rec- ognize and generate humorous content. In the artificial intelligence community, research on humor has been progressing slowly but steadily. As an effort to boost research and spur new ideas in this challenging area, we created a competitive task for automatically assessing humor in edited news headlines. Like other AI tasks, automatic humor recognition depends on labeled data. Nearly all existing humor datasets are annotated to study the binary task of whether a piece of text is funny (Mihalcea and Strapparava, 2005; Kiddon and Brun, 2011; Bertero and Fung, 2016; Raz, 2012; Filatova, 2012; Zhang and Liu, 2014; Reyes et al., 2012; Barbieri and Saggion, 2014). Such categorical data does not capture the non-binary character of humor, which makes it difficult to develop models that can predict a level of funniness. Humor occurs in various intensities, and certain jokes are much funnier than others, including the supposedly funniest joke in the world (Wiseman, 2011). A system’s ability to assess the degree of humor makes it useful in various applications, such as in humor generation where such a system can be used in a generate-and-test scheme to generate many potentially humorous texts and rank them by funniness, for example, to automatically fill in the blanks in Mad Libs R⃝ for humorous effects (Hossain et al., 2017; Garimella et al., 2020). For our SemEval task, we provided a dataset that contains news headlines with short edits applied to them to make them humorous (see Table 1). This dataset was annotated as described in Hossain et al. (2019) using Amazon Mechanical Turk, where qualified human workers edited headlines to make them funny and the quality of humor in these headlines was assessed by a separate set of …

Content truncated to 500 words