CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1823 CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning Bill Yuchen Lin♥ Wangchunshu Zhou♥ Ming Shen♥ Pei Zhou♥ Chandra Bhagavatula♠ Yejin Choi♠♦ Xiang Ren♥ ♥University of Southern California ♠Allen Institute for Artificial Intelligence ♦Paul G. Allen School of Computer Science & Engineering, University of Washington {yuchen.lin,xiangren}@usc.edu,{chandrab,yejinc}@allenai.org Abstract Recently, large-scale pretrained language mod- els have demonstrated impressive performance on several commonsense-reasoning bench- mark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text gen- eration task, COMMONGEN associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense rea- soning. Given a set of common concepts (e.g., {dog, frisbee, catch, throw}); the task is to gen- erate a coherent sentence describing an every- day scenario using these concepts (e.g., “a man throws a frisbee and his dog catches it”). The COMMONGEN task is challenging be- cause it inherently requires 1) relational rea- soning with background commonsense knowl- edge, and 2) compositional generalization abil- ity to work on unseen concept combinations. Our dataset, constructed through a combina- tion of crowdsourced and existing caption cor- pora, consists of 77k commonsense descrip- tions over 35k unique concept-sets. Exper- iments show that there is a large gap be- tween state-of-the-art text generation models (e.g., T5) and human performance (31.6% v.s. 63.5% in SPICE metric). Furthermore, we demonstrate that the learned generative com- monsense reasoning capability can be trans- ferred to improve downstream tasks such as CommonsenseQA (76.9% to 78.4 in dev accu- racy) by generating additional context. 1 Introduction Commonsense reasoning, the ability to make ac- ceptable and logical assumptions about ordinary scenes in our daily life, has long been acknowl- edged as a critical bottleneck of artificial intelli- gence and natural language processing (Davis and Marcus, 2015). Most recent commonsense rea- soning challenges, such as CommonsenseQA (Tal- dog, frisbee, catch, throw - A dog leaps to catch a thrown frisbee. - The dog catches the frisbee when the boy throws it. - A man throws away his dog 's favorite frisbee expecting him to catch it in the air. Expected Output: everyday scenarios covering all given concepts. [Humans] GPT2: A dog throws a frisbee at a football player. UniLM: Two dogs are throwing frisbees at each other . BART: A dog throws a frisbee and a dog catches it. T5: dog catches a frisbee and throws it to a dog [Machines] exercise | rope | wall | tie | wave - A man in a gym exercises by waving ropes tied to a wall. - The gym owner decided to tie a rope to the wall so people could make a wave in it for exercise. Concept-Set: [Humans] GPT2: A woman is tied up in a rope and swinging a wave at a wall. UniLM: A man with a rope and tie is doing some exercise …

Content truncated to 500 words