Can Language Models Make Fun? A Case Study in Chinese Comical Crosstalk

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 688–714 July 9-14, 2023 ©2023 Association for Computational Linguistics Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest Jack Hessel† Ana Marasovi´c⋄ Jena D. Hwang† Lillian Lee◦ Jeff Da‡ Rowan Zellers• Robert Mankoff▲ Yejin Choi†‡ † The Allen Institute for AI ⋄ University of Utah ◦ Cornell University •OpenAI ‡ University of Washington ▲ Air Mail and Cartoon Collections jackh@allenai.org ana.marasovic@utah.edu jenah@allenai.org llee@cs.cornell.edu {jzda,rowanz}@cs.washington.edu bob@bobmankoff.com yejin@cs.washington.edu Abstract Large neural networks can now generate jokes, but do they really “understand” humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Con- test: matching a joke to a cartoon, identify- ing a winning caption, and explaining why a winning caption is funny. These tasks en- capsulate progressively more sophisticated as- pects of “understanding” a cartoon; key ele- ments are the complex, often surprising rela- tionships between images and captions and the frequent inclusion of indirect and playful allu- sions to human experience and culture. We in- vestigate both multimodal and language-only models: the former are challenged with the car- toon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored ex- planations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the im- age’s locations/entities, what’s unusual in the scene, and an explanation of the joke. 1 Introduction Humor can be dissected, as a frog can, but the thing dies in the process and the innards are discouraging to any but the pure scientific mind. – White, E. B. (1941) Each week, The New Yorker publishes a un- captioned cartoon image, inviting readers to sub- mit their funniest English-language caption for it. Editors choose three finalists from sometimes thou- sands of submissions; then, readers vote to pick Human-authored: When drinking coffee or tea, people often add cream, and may ask others to pass it if it's on the other side of a table. But here, the mugs are huge, so instead of asking for a small cup of cream, they are asking for the entire cow, which is the appropriately-sized cream dispenser for these huge drinks. From Description (5-shot GPT 3.5): "Pass the cow" is an example of a non sequitur, something that looks like a logical thing to say, but doesn't make sense in context. The humor in this cartoon comes from the large size of the coffee mugs: they are so large that they resemble buckets rather than mugs, thus making the request to "pass the cow" almost reasonable. From Pixels (OFA + T5-11B): The joke is …

Content truncated to 500 words