Hitachi at SemEval-2020 Task 7: Stacking at Scale with Heterogeneous Language Models for Humor Recognition

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 791–803 Barcelona, Spain (Online), December 12, 2020. 791 Hitachi at SemEval-2020 Task 7: Stacking at Scale with Heterogeneous Language Models for Humor Recognition Terufumi Morishita∗, Gaku Morio*, Hiroaki Ozaki and Toshinori Miyoshi Hitachi, Ltd. Resarch and Development Group Kokubunji, Tokyo, Japan {terufumi.morishita.wp, gaku.morio.vn, hiroaki.ozaki.yu, toshinori.miyoshi.pd}@hitachi.com Abstract This paper describes the winning system for SemEval-2020 task 7: Assessing Humor in Edited News Headlines. Our strategy is Stacking at Scale (SaS) with heterogeneous pre-trained lan- guage models (PLMs) such as BERT and GPT-2. SaS first performs fine-tuning on numbers of PLMs with various hyperparameters and then applies a powerful stacking ensemble on top of the fine-tuned PLMs. Our experimental results show that SaS outperforms a naive average ensemble, leveraging weaker PLMs as well as high-performing PLMs. Interestingly, the results show that SaS captured non-funny semantics. Consequently, the system was ranked 1st in all subtasks by significant margins compared with other systems. 1 Introduction The recognition of humor in text has been receiving much attention (Barbieri and Saggion, 2014; Hossain et al., 2019). Accordingly, SemEval-2020 task 7, Assessing Humor in Edited News Headlines (Hossain et al., 2020a), which aims at automatically recognizing humor in hand-edited news headlines, was held with two subtasks: Subtask 1, which aims at predicting a funny score for an edited news headline, and Subtask 2, which aims at predicting the funnier headline of two given edited headlines. In this paper, we pursue humor recognition with a large-scale stacking ensemble (hereafter Stacking at Scale or SaS), by leveraging pre-trained language models (PLMs). SaS is based on an ensemble method where a meta-estimator is trained to predict labels from the outputs of base models, finding the best combinations of the base models (Wolpert, 1992). Hence, there are two steps in SaS: (i) fine-tuning numbers of heterogeneous PLMs, including BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), RoBERTa (Liu et al., 2019), Transformer-XL (Dai et al., 2019), XLNet (Yang et al., 2019), and XLM (Lample and Conneau, 2019), with various hyperparameters, obtaining rich and diverse models, and (ii) training a meta-estimator on top of these PLMs. Our experiments, fusing up to 1750 PLMs in total, indicate that SaS successfully leverages weaker PLMs as well as high-performing PLMs. Consequently, our system is ranked 1st on both subtasks with significant margins to others. Interestingly, analyses show that SaS learned (relatively) non-funny semantics while still struggling to understand the funniest semantics. To the best of our knowledge, this is the first experiment that involves thousands of diverse of PLMs, revealing the current strengths and limitations of PLMs in automatic humor recognition. We also provide useful insights obtained from rich analyses. 2 Background Work related to humor recognition has been done in recent years (Khodak et al., 2018; Barbieri and Saggion, 2014; Reyes et al., 2012). Khodak et al. (2018) introduced a large-scale annotated corpus of sarcasm and provided baseline systems for sarcasm detection. Barbieri and Saggion (2014) widely ∗Contributed equally. This work is …

Content truncated to 500 words