COPA-SSE: Semi-structured Explanations for Commonsense Reasoning

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3994–4000 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3994 COPA-SSE: Semi-structured Explanations for Commonsense Reasoning Ana Brassard,1, 2 Benjamin Heinzerling,1, 2 Pride Kavumba,2, 1 Kentaro Inui2, 1 1Riken AIP, 2Tohoku NLP Lab {ana.brassard, benjamin.heinzerling}@riken.jp, kavumba.pride.q2@dc.tohoku.ac.jp, inui@tohoku.ac.jp Abstract We present Semi-Structured Explanations for COPA (COPA-SSE), a new crowdsourced dataset of 9,747 semi-structured, En- glish common sense explanations for Choice of Plausible Alternatives (COPA) questions. The explanations are formatted as a set of triple-like common sense statements with ConceptNet relations but freely written concepts. This semi-structured for- mat strikes a balance between the high quality but low coverage of structured data and the lower quality but high coverage of free-form crowdsourcing. Each explanation also includes a set of human-given quality ratings. With their familiar format, the explanations are geared towards commonsense reasoners operating on knowledge graphs and serve as a starting point for ongoing work on improving such systems. The dataset is available at https://github.com/a-brassard/copa-sse. Keywords: Collaborative Resource Construction & Crowdsourcing, Corpus (Creation, Annotation, etc.), Knowledge Discov- ery/Representation, Question Answering 1. Introduction While there are many datasets for question answer- ing and commonsense reasoning (Rogers et al., 2021), models are known to exploit shortcuts such as super- ficial cues in these datasets, which leads to artificially high evaluation scores (Gururangan et al., 2018). One way to ensure models are reasoning as intended is to re- quire explanations for their predictions (Bowman and Dahl, 2021). A prominent example of such a set- ting is the Commonsense Explanations Dataset (CoS- E) (Rajani et al., 2019), which provides crowdsourced justifications of the correct answers expressed in free text. While free-form crowdsourcing allows represent- ing natural and diverse human reasoning, quality con- trol is notoriously difficult (Daniel et al., 2018). At the other end of the spectrum are explanations that are fully grounded in a knowledge graph (KG), i.e., each ele- ment of the explanation corresponds to a node or edge in a KG. However, this structured approach is limited by the coverage of the KG, i.e., the explanation will be sub-optimal or impossible when the situation to ex- plain is not covered by the KG. Here, we adopt a semi- structured approach aiming to combine the best of both worlds—the coverage potential of open-ended crowd- sourcing and quality control of structured data. Specifically, we introduce Semi-Structured Expla- nations for COPA (COPA-SSE), a new explana- tion dataset for the Choice of Plausible Alternatives (COPA) dataset (Roemmele et al., 2011).1 Each expla- nation consists of a set of English statements, which, in turn, consist of a head text, a selected predicate, and tail text, mimicking ConceptNet (Speer et al., 2017) triples (Figure 1). The head and tail texts are free-form, al- lowing an open concept inventory. Each explanation 1We use Balanced COPA (Kavumba et al., 2019), a super- set of COPA. Cracked tiles is made of sharp edges. Sharp edges is capable of cutting foot. Cutting is capable …

Content truncated to 500 words