“President Vows to Cut <Taxes> Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines

Proceedings of NAACL-HLT 2019, pages 133–142 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 133 “President Vows to Cut Taxes Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines Nabil Hossain Dept. Computer Science University of Rochester Rochester, NY nhossain@cs.rochester.edu John Krumm and Michael Gamon Microsoft Research AI Microsoft Corporation Redmond, WA {jckrumm,mgamon}@microsoft.com Abstract We introduce, release, and analyze a new dataset, called Humicroedit, for research in computational humor. Our publicly available data consists of regular English news head- lines paired with versions of the same head- lines that contain simple replacement edits de- signed to make them funny. We carefully curated crowdsourced editors to create funny headlines and judges to score a to a total of 15,095 edited headlines, with five judges per headline. The simple edits, usually just a single word replacement, mean we can apply straightforward analysis techniques to deter- mine what makes our edited headlines humor- ous. We show how the data support classic the- ories of humor, such as incongruity, superior- ity, and setup/punchline. Finally, we develop baseline classifiers that can predict whether or not an edited headline is funny, which is a first step toward automatically generating humor- ous headlines as an approach to creating topi- cal humor. 1 Introduction Humor detection and generation continue to be challenging AI problems. While there have been some advances in automatic humor recognition (Khodak et al., 2017; Davidov et al., 2010; Bar- bieri and Saggion, 2014; Reyes et al., 2012; Cat- tle and Ma, 2018; Bertero and Fung, 2016; Yang et al., 2015), computerized humor generation has seen less progress (Binsted et al., 1997; Stock and Strapparava, 2003; Petrovi´c and Matthews, 2013). This is not surprising, given that humor involves in-depth world-knowledge, common sense, and the ability to perceive relationships across enti- ties and objects at various levels of understanding. Even humans often fail at being funny or recog- nizing humor. A big hindrance to progress on humor re- search is the scarcity of public datasets. Further- (a) The Headline Editing Task. (b) The Headline Grading Task. Figure 1: Snapshots of the headline editing and grading interfaces. Only the underlined tokens are replaceable. more, the existing datasets address specific hu- mor templates, such as funny one-liners (Mihal- cea and Strapparava, 2006) and filling in Mad Libs R⃝ (Hossain et al., 2017). Creating a humor corpus is non-trivial, however, because it requires (i) human annotation, and (ii) a clear definition of humor to achieve good inter-annotator agreement. We introduce Humicroedit, a novel dataset for research in computational humor. First, we collect original news headlines from news media posted on Reddit (reddit.com). Then, we qualify ex- pert annotators from Amazon Mechanical Turk (mturk.com) to (i) generate humor by applying small edits to these headlines, and to (ii) judge the humor in these edits. Our resulting dataset con- tains 15,095 edited news headlines and their nu- merically assessed humor. Screenshots of our two annotation tasks are shown in Figure 1, and Ta- ble …

Content truncated to 500 words