Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 24–39 July 13-14, 2023 ©2023 Association for Computational Linguistics When Truth Matters – Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs) Reto Gubelmann University of St. Gallen reto.gubelmann@unisg.ch Aikaterini-Lida Kalouli CIS - LMU Munich katerina.kalouli@hotmail.com Christina Niklaus University of St. Gallen christina.niklaus@unisg.ch Siegfried Handschuh University of St. Gallen siegfried.handschuh@unisg.ch Abstract In this paper, we focus on the ability of large language models (LLMs) to accommodate dif- ferent pragmatic sentence types, such as ques- tions, commands, as well as sentence fragments for natural language inference (NLI). On the commonly used notion of logical inference, nothing can be inferred from a question, a com- mand, or an incomprehensible sentence frag- ment. We find MNLI, arguably the most impor- tant NLI dataset, and hence models fine-tuned on this dataset, insensitive to this fact. Using a symbolic semantic parser, we develop and make publicly available, fine-tuning datasets designed specifically to address this issue, with promising results. We also make a first explo- ration of ChatGPT’s concept of entailment. 1 Introduction: “I didn’t say that!’ Committing oneself to the truth of a certain claim always implies or suggests one’s commitment to the truth of a number of other claims, and it pre- cludes one’s commitment to the truth of a second set of claims. This is the essence of the logical no- tion of entailment (here used synonymously with inference) and contradiction. For instance, some- body who claims “Loral did harm national security” is also committed to the less specific claim “Some- thing or somebody did harm national security”. The concept of inference is itself quite complex. As Gubelmann et al. (2022) detail, valid inferences can be divided into deductively valid inferences, where it is not logically (see Plantinga 1974) possi- ble that the premise is true while the conclusion is false, and inductively valid inferences (also called abductions), where it is possible that the premise is true while the conclusion is false, but where the truth of the premise is in general a good reason for the truth of the conclusion. For two utterances to be able to (deductively or inductively) entail or contradict each other, they have to be of the correct pragmatic category. While assertions or claims are able to stand in these logi- cal relationships, the same does not hold for other pragmatic kinds of utterances, such as questions or commands, as they do not involve making a claim that could be true or false and hence commit their author to the truth of certain claims that could then entail or contradict other claims. For instance, uttering (P) in example (1) does not commit the speaker to Loral’s harming of national security – she’s simply asking a question, not making a claim. Hence if, after uttering (P) from (1), somebody replies with “So you claim that somebody did harm national security”, the appropriate response would be “I didn’t say that!”. Neither …

Content truncated to 500 words