RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1504–1515 August 1–6, 2021. ©2021 Association for Computational Linguistics 1504 RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge Bill Yuchen Lin Ziyi Wu Yichi Yang Dong-Ho Lee Xiang Ren {yuchen.lin,ziyiwu,yichiyan,dongho.lee,xiangren}@usc.edu Department of Computer Science and Information Sciences Institute, University of Southern California Abstract Question: I have five fingers but I am not alive. What am I? Answer: a glove. Answering such a riddle-style question is a challenging cognitive process, in that it re- quires complex commonsense reasoning abil- ities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural lan- guage understanding (NLU). However, there is currently no dataset aiming to test these abilities. In this paper, we present RIDDLE- SENSE1, a new multiple-choice question an- swering task, which comes with the first large dataset (5.7k examples) for answering riddle- style commonsense questions. We systemat- ically evaluate a wide range of models over the RIDDLESENSE challenge, and point out that there is a large gap between the best- supervised model and human performance — suggesting intriguing future research in the di- rection of higher-order commonsense reason- ing and linguistic creativity towards building advanced NLU systems. 1 Introduction “ The essence of a riddle is to express true facts under impossible combinations.” — Aristotle, Poetics (350 BCE) A riddle is a puzzling question about concepts in our everyday life. For example, a riddle might ask “My life can be measured in hours. I serve by be- ing devoured. Thin, I am quick. Fat, I am slow. Wind is my foe. What am I?” The correct answer “candle,” is reached by considering a collection of commonsense knowledge: a candle can be lit and burns for a few hours; a candle’s life depends upon its diameter; wind can extinguish candles, etc. It is believed that the riddle is one of the earli- est forms of oral literature, which can be seen as 1https://inklab.usc.edu/RiddleSense/ My life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm What home entertainment equipment requires cable? (A) radio shack (B) substation (C) cabinet (D) television (E) desk CSQA RiddleSense I have five fingers, but I am not alive. What am I? (A) piano (B) computer (C) glove (D) claw (E) hand D B C D VS Figure 1: The top example is a trivial commonsense question from the CommonsenseQA (Talmor et al., 2019) dataset. The two bottom examples are from our proposed RIDDLESENSE challenge. The right-bottom question is a descriptive riddle that implies multiple commonsense facts about candle, and it needs under- standing of figurative language such as metaphor; The left-bottom one additionally needs counterfactual rea- soning ability to address the ‘but-no’ cues. These riddle-style commonsense questions require NLU sys- tems to have higher-order reasoning skills with the un- derstanding of creative language …

Content truncated to 500 words