SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 49–57, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor Peter Potash, Alexey Romanov, Anna Rumshisky University of Massachusetts Lowell Department of Computer Science {ppotash,aromanov,arum}@cs.uml.edu Abstract This paper describes a new shared task for humor understanding that attempts to es- chew the ubiquitous binary approach to humor detection and focus on comparative humor ranking instead. The task is based on a new dataset of funny tweets posted in response to shared hashtags, collected from the ‘Hashtag Wars’ segment of the TV show @midnight. The results are eval- uated in two subtasks that require the par- ticipants to generate either the correct pair- wise comparisons of tweets (subtask A), or the correct ranking of the tweets (subtask B) in terms of how funny they are. 7 teams participated in subtask A, and 5 teams par- ticipated in subtask B. The best accuracy in subtask A was 0.675. The best (lowest) rank edit distance for subtask B was 0.872. 1 Introduction Most work on humor detection approaches the problem as binary classification: humor or not hu- mor. While this is a reasonable initial step, in practice humor is continuous, so we believe it is interesting to evaluate different degrees of humor, particularly as it relates to a given person’s sense of humor. To further such research, we propose a dataset based on humorous responses submitted to a Comedy Central TV show, allowing for com- putational approaches to comparative humor rank- ing. Debuting in Fall 2013, the Comedy Central show @midnight1 is a late-night “game-show” that presents a modern outlook on current events by focusing on content from social media. The show’s contestants (generally professional come- dians or actors) are awarded points based on how 1http://www.cc.com/shows/-midnight funny their answers are. The segment of the show that best illustrates this attitude is the Hash- tag Wars (HW). Every episode the show’s host proposes a topic in the form of a hashtag, and the show’s contestants must provide tweets that would have this hashtag. Viewers are encouraged to tweet their own responses. From the viewers’ tweets, we are able to apply labels that determine how relatively humorous the show finds a given tweet. Because of the contest’s format, it provides an adequate method for addressing the selection bias (Heckman, 1979) often present in machine learn- ing techniques (Zadrozny, 2004). Since each tweet is intended for the same hashtag, each tweet is ef- fectively drawn from the same sample distribution. Consequently, tweets are seen not as humor/non- humor, but rather varying degrees of wit and clev- erness. Moreover, given the subjective nature of humor, labels in the dataset are only “gold” with respect to the show’s sense of humor. This concept becomes more grounded when considering the use of supervised systems for the dataset. The idea of the dataset is to learn to character- ize the sense of humor represented in this show. …

Content truncated to 500 words