Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020–4026 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4020 Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation Cunxiang Wang1,4, Shuailong Liang2, Yue Zhang1, Xiaonan Li3 and Tian Gao4 1School of Engineering, Westlake University, China 2Singapore University of Technology and Design, Singapore 3School of Computer Science and Technology, Xidian University, China 4College of Computer Science and Technology, Zhejiang University, China wangcunxiang@westlake.edu.cn, yue.zhang@wias.org.cn shuailong liang@mymail.sutd.edu.sg lixiaonan xdu@outlook.com, gaotian@zju.edu.cn Abstract Introducing common sense to natural language understanding systems has received increasing research attention. It remains a fundamental question on how to evaluate whether a system has a sense making capability. Existing bench- marks measures commonsense knowledge in- directly and without explanation. In this pa- per, we release a benchmark to directly test whether a system can differentiate natural lan- guage statements that make sense from those that do not make sense. In addition, a system is asked to identify the most crucial reason why a statement does not make sense. We evalu- ate models trained over large-scale language modeling tasks as well as human performance, showing that there are different challenges for system sense making. 1 Introduction Natural Language Understanding (NLU) has re- ceived increasing research attention in recent years. With language models trained on large cor- pora (Peters et al., 2018; Devlin et al., 2018), al- gorithms show better performance than humans on some benchmarks (Group, 2017; Devlin et al., 2018). Compared to humans, however, most end- to-end trained systems are rather weak on common sense. For example, it is straightforward for a hu- man to understand that someone can put a turkey into a fridge but he can never put an elephant into a fridge with basic commonsense reasoning, but it can be non-trivial for a system to tell the differ- ence. Arguably, commonsense reasoning should be a central capability in a practical NLU system (Davis, 2017); it is, therefore, important to be able to evaluate how well a model can do for sense making. Existing datasets test common sense indirectly through tasks that require extra knowledge, such Figure 1: Samples of our dataset as co-reference resolution (Levesque et al., 2012; Morgenstern and Ortiz, 2015), subsequent event prediction (Roemmele et al., 2011; Zhang et al., 2017; Zellers et al., 2018), or reading comprehen- sion (Mostafazadeh et al., 2016; Ostermann et al., 2018b). They verify whether a system is equipped with common sense by testing whether it can give a correct answer where the input does not contain such knowledge. However, there are two limita- tions to such benchmarks. First, they do not give a direct metric to quantitatively measure sense mak- ing capability. Second, they do not explicitly iden- tify the key factors required in a sense making pro- cess. We address these issues by creating a testset for direct sense making. As shown in Figure 1, the 4021 benchmark consists of two subtasks. The first …

Content truncated to 500 words