HAHA 2019 Dataset: A Corpus for Humor Analysis in Spanish

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5106–5112 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5106 HAHA 2019 Dataset: A Corpus for Humor Analysis in Spanish Luis ChiruzzoR, Santiago CastroM R, Aiala RosáR RUniversidad de la República, Uruguay MUniversity of Michigan, USA luischir@fing.edu.uy, sacastro@umich.edu, aialar@fing.edu.uy Abstract This paper presents the development of a corpus of 30,000 Spanish tweets that were crowd-annotated with humor value and funniness score. The corpus contains approximately 38.6% intended-humor tweets with a 2.04/5 average funniness score. It has been used in an automatic humor recognition and analysis competition, obtaining encouraging results from the participants. Keywords: Humor, Computational Humor, Humor Detection, Humor Analysis, Natural Language Processing 1. Introduction Within the subfield of Computational Humor, there have been several works that have built resources to study dif- ferent forms of Humor in texts (Mihalcea and Strapparava, 2005; Sjöbergh and Araki, 2007; Yang et al., 2015; Potash et al., 2017). For the case of the Spanish language, (Castro et al., 2016) created a first humor dataset, in which several issues were subsequently addressed by (Castro et al., 2018). Both of these works extracted tweets from both live random samples and selected accounts and had them annotated ac- cording to intended humor (binary), and a funniness score from one to five for the latter case. The dataset created in (Castro et al., 2018) was used by Castro et al. (2018) in the context of the HAHA 2018 competition for Humor Detection and Funniness Average Prediction. However, the dataset presented in (Castro et al., 2018) still presents some issues. First, there are many duplicate tweets (around 6%), which only differ in their format or spacing, but the text is essentially the same. This is a potential prob- lem with the quality of the data as they may have different annotations. Second, we believe there is some room for improvement for the inter-annotator agreement, which is 0.57 in Krippendorff’s alpha value (Krippendorff, 2012). Finally, there is a class unbalance that should be tackled as it does not represent a sample from the reality (as tweets are picked from different sources) and complicates training and evaluation. In this work, we build on top of (Castro et al., 2018), tackling the mentioned issues and creating a new dataset. We gather more tweets following a similar crowdsourcing annotation procedure but tackling some of the issues to increase the agreement score and to have a more balanced dataset, and we put it together along with the tweets from their dataset by removing the duplicate tweets and merging the annota- tions. Additionally, during the dataset annotation, we asked the annotators to tell if they considered the tweet text to be offensive or not. We find interesting to study how hu- mor plays along with hate speech. This dataset was used (Chiruzzo et al., 2019) in the context of the HAHA 2019 competition, hosted at IberLEF, for Humor Detection and Funniness Score Prediction. 2. Related Work …

Content truncated to 500 words