Development and Validation of a Corpus for Machine Humor Comprehension

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1346–1352 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1346 Development and Validation of a Corpus for Machine Humor Comprehension Yuen-Hsien Tseng, Wun-Syuan Wu, Chia-Yueh Chang, Hsueh-Chih Chen, Wei-Lun Hsu National Taiwan Normal University No. 162, Sec. 1, Heping East Rd., Taipei 10610, Taiwan MOST AI Biomedical Research Center {samtseng, 40420213l, 80701003E, chcjyh, 60715004e}@ntnu.edu.tw Abstract This work developed a Chinese humor corpus containing 3,365 jokes collected from over 40 sources. Each joke was labeled with five levels of funniness, eight skill sets of humor, and six dimensions of intent by only one annotator. To validate the manual labels, we trained SVM (Support Vector Machine) and BERT (Bidirectional Encoder Representations from Transformers) with half of the corpus (labeled by one annotator) to predict the skill and intent labels of the other half (labeled by the other annotator). Based on two assumptions that a valid manually labeled corpus should follow, our results showed the validity for the skill and intent labels. As to the funniness label, the validation results showed that the correlation between the corpus label and user feedback rating is marginal, which implies that the funniness level is a harder annotation problem to be solved. The contribution of this work is two folds: 1) a Chinese humor corpus is developed with labels of humor skills, intents, and funniness, which allows machines to learn more intricate humor framing, effect, and amusing level to predict and respond in proper context (https://github.com/SamTseng/Chinese_Humor_MultiLabeled). 2) An approach to verify whether a minimum human labeled corpus is valid or not, which facilitates the validation of low-resource corpora. Keywords: Humor Corpus, Humor Framing, Humor Intent, Corpus Validation, Multi-label Classification, Traditional Chinese 1. Introduction Humor is an important element in inter-personal communication. In commercial applications, a humorous communication process can often dispel user complaints (Bellegarda, 2014; Binsted, 1995). In education, previous research (Bryant & Zillmann, 1989; Mcghee & Frank, 2014) has found that proper use of humor in the classroom can attract students' attention, improve in-class interaction, and help students learn better with fun. In more high-stake situations such as in delivering a speech in public, humor can help speakers reduce anxiety and improve their performance. In addition, advertising, entertainment, and other commercial sectors are also the domains of humor applications. As human-dialogue systems, chatbots, or conversational user interface become versatile in this resurgent AI (Artificial Intelligence) age, the introduction of humor to the human-machine communication become important for the above reasons. This requires the system to recognize human humor or generate humorous utterance. To start building such a system, a humor corpus, commonly in the form of a joke collection, is needed, especially a manual labeled corpus for machines to learn to identify or even to generate humorous dialogues. There have been a number of joke corpora collected in past studies. However, most corpora are just collections of joke stories, hilarious snippets, puns, or one-liners. Some with negative examples …

Content truncated to 500 words