“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 407–417 May 2-6, 2023 ©2023 Association for Computational Linguistics John is 50 years old, can his son be 65? Evaluating NLP Models’ Understanding of Feasibility Himanshu Gupta Neeraj Varshney Swaroop Mishra Kuntal Kumar Pal Saurabh Arjun Sawant Kevin Scaria Siddharth Goyal Chitta Baral Arizona State University Abstract In current NLP research, large-scale language models and their abilities are widely being dis- cussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question- answering dataset involving binary classifi- cation (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 strug- gle to answer the feasibility questions correctly. Specifically, on MCQ and BCQ questions, GPT- 3 achieves an accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. We also evaluate models by pro- viding relevant knowledge statements required to answer the question. We find that the ad- ditional knowledge leads to a 7% gain in per- formance, but the overall performance still re- mains low. These results make one wonder how much commonsense knowledge about action feasibility is encoded in state-of-the-art models and how well they can reason about it. 1 1 Introduction Commonsense reasoning has been a key aspect of AI since its advent in the 1950s. It is closely associ- ated with reasoning about actions and their effects, which is considered a big challenge, especially for deep learning-based AI approaches and language models (LeCun, 2022; Dalvi et al., 2018; Banerjee et al., 2020). While several datasets have been de- veloped to evaluate large-scale language models, in this paper, we propose a dataset focused on reason- ing about actions and their effects; specifically, the ability to reason if an action or its effect is feasible. 1Dataset, baseline approaches, and instruction-tuned mod- eling approaches are freely available at https://github. com/kevinscaria/feasibilityQA Figure 1: Illustrating a binary classification (BCQ) and a multiple choice question (MCQ) from FeasibilityQA. The correct answer options (False in BCQ and (45, 37) in MCQ) are highlighted in bold. Figure 1 illustrates an example of our dataset; given the information “Sam gave 50 dollars to the shopkeeper to buy a book and the shopkeeper re- turned some money”, it is not possible to compute the exact price of the book; however, it can be estab- lished that the feasible price of the book is less than 50 since the shopkeeper returned some money. We often come across such questions in our daily life and find it trivial to reason about them. Therefore, in order to develop NLP systems that can reliably reason about real-world situations, it is important to evaluate their understanding of feasibility. Recently, many datasets have been created that …

Content truncated to 500 words