A Method for Building a Commonsense Inference Dataset based on Basic Events

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2450–2460, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2450 A Method for Building a Commonsense Inference Dataset based on Basic Events Kazumasa Omura Daisuke Kawahara∗ Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan {omura, dk, kuro}@nlp.ist.i.kyoto-u.ac.jp Sadao Kurohashi Abstract We present a scalable, low-bias, and low-cost method for building a commonsense infer- ence dataset that combines automatic extrac- tion from a corpus and crowdsourcing. Each problem is a multiple-choice question that asks contingency between basic events. We ap- plied the proposed method to a Japanese cor- pus and acquired 104k problems. While hu- mans can solve the resulting problems with high accuracy (88.9%), the accuracy of a high- performance transfer learning model is reason- ably low (76.0%). We also confirmed through dataset analysis that the resulting dataset con- tains low bias. We released the datatset to fa- cilitate language understanding research.1 1 Introduction Along with the progress of deep learning, there have been many studies that consider task settings and build their datasets for training/evaluating lan- guage understanding ability by computers (Wang et al., 2019b,a). Language understanding by computers requires two types of knowledge: knowledge of language (meaning of words, syntax, and so forth) and knowl- edge of our world and society beyond language. The former problem of acquiring linguistic knowledge has been solved to a large extent by general-purpose language models, such as BERT (Devlin et al., 2019), which are pre-trained using a large corpus. It is now possible to represent the meaning of a word as a vector according to its con- text. Fine-tuning based on these vectors has made natural language inference, paraphrase recognition, and question answering without requiring deep in- ference as accurate as humans. On the other hand, there are still many problems with acquiring knowledge beyond language. Actu- ∗Current affiliation is Waseda University. 1http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?JCID ally it is open-ended, and we had better start with fundamental knowledge, i.e., commonsense. Still, it is not easy to focus on commonsense, guarantee- ing some generality as commonsense. There have been some approaches to guarantee such generality. SWAG (Zellers et al., 2018), for example, focuses on knowledge about daily events that can be visually perceived. This method greatly limits the range of commonsense that can be ac- quired. CommonsenseQA (Talmor et al., 2019) is based on the basic vocabulary that is covered by ConceptNet (Speer et al., 2017), which is one of the largest commonsense knowledge bases. This pre- vents the scalability, generating only 12k problems from the whole data of ConceptNet. Another important point is that biases in build- ing datasets must be reduced as much as possi- ble. In the above two approaches, question or dis- tractor sentences were created automatically or by crowdsourcing. This causes generation biases of language models or produces certain patterns (an- notation artifacts) by crowdsourced writing (Guru- rangan et al., 2018). We use a text corpus to solve these problems. We propose a method to …

Content truncated to 500 words