To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning.

Proceedings of the 29th International Conference on Computational Linguistics, pages 1708–1717 October 12–17, 2022. 1708 To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. Yitian Li1,2†, Jidong Tian1,2†, Wenqing Chen1,2, Caoyun Fan1,2, Hao He1,2‡ and Yaohui Jin1,2 1MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2State Key Lab of Advanced Optical Communication System and Network, Shanghai Jiao Tong University {yitian_li, frank92, wenqingchen, fcy3649, hehao, jinyh}@sjtu.edu.cn Abstract Reasoning and knowledge-related skills are considered as two fundamental skills for natu- ral language understanding (NLU) tasks such as machine reading comprehension (MRC) and natural language inference (NLI). However, it is not clear to what extent an NLU task defined on a dataset correlates to a specific NLU skill. On the one hand, evaluating the correlation re- quires an understanding of the significance of the NLU skill in a dataset. Significance judges whether a dataset includes sufficient material to help the model master this skill. On the other hand, it is also necessary to evaluate the dependence of the task on the NLU skill. De- pendence is a measure of how much the task defined on a dataset depends on the skill. In this paper, we propose a systematic method to diagnose the correlations between an NLU dataset and a specific skill, and then take a fun- damental reasoning skill, logical reasoning, as an example for analysis. The method adopts a qualitative indicator to indicate the significance while adopting a quantitative indicator to mea- sure the dependence. We perform diagnosis on 8 MRC datasets (including two types) and 3 NLI datasets and acquire intuitively reason- able results. We then perform the analysis to further understand the results and the proposed indicators. Based on the analysis, although the diagnostic method has some limitations, it is still an effective method to perform a basic di- agnosis of the correlation between the dataset and logical reasoning skill, which also can be generalized to other NLU skills. 1 Introduction Machine reading comprehension (MRC) and nat- ural language inference (NLI) are used to bench- mark natural language understanding (NLU) capa- bilities. Although a large number of NLU-related † These authors contributed equally. ‡ Corresponding author. datasets have been proposed (Yang et al., 2018; Yatskar, 2019), it is hard to evaluate correlations between the dataset and NLU-related skills due to the lack of benchmark method (Richardson et al., 2020), which may create an obstacle to choose ap- propriate dataset for specific skill training. Accord- ing to a widely accepted discourse comprehension theory, construction-integration model (Kintsch, 1991), there are two processes for humans to un- derstand language: 1) understanding the concepts and discourses in the text and building their rela- tionships with the real world; 2) synthesizing these concepts and discourses to form a consistent under- standing. These two processes mainly correspond to two types of skills in NLU: Knowledge-related skills and reasoning skills (Baral et al., 2020; Bo- ratko et al., 2018; Tian et al., 2021). Previous …

Content truncated to 500 words