Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6031 - 6037 July 10-15, 2022 ©2022 Association for Computational Linguistics Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference Em¯ıls Kadik, is, Vaibhav Srivastav, and Roman Klinger Institut für Maschinelle Sprachverarbeitung University of Stuttgart Pfaffenwaldring 5b, 70569 Stuttgart {emils.kadikis,vaibhav.srivastav,klinger@ims.uni-stuttgart.de} Abstract The task of abductive natural language infer- ence (αNLI), to decide which hypothesis is the more likely explanation for a set of obser- vations, is a particularly difficult type of NLI. Instead of just determining a causal relation- ship, it requires common sense to also eval- uate how reasonable an explanation is. All recent competitive systems build on top of con- textualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particu- lar NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this prac- tical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the αNLI task when just comparing sentence embeddings with cosine similarity to what the performance that is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity ap- proach correlates strongly with the accuracy of the classification approach with a Pearson cor- relation coefficient of 0.65. Since the similarity computation is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection. 1 Introduction Abduction is a type of reasoning that infers an ex- planation for some observations (Peirce, 1931). It is a particularly challenging type of inference; as opposed to deduction and induction, which derive the conclusion from only the information present in the observations, abduction requires making as- sumptions about an implicit context beyond just the given observations. Abductive reasoning is there- fore at the core of the way humans understand the world and how world knowledge is involved. Abductive reasoning in the natural language do- main has been introduced by Bhagavatula et al. (2020) who defined the abductive natural language inference (αNLI) task. In it, we are given four sentences – two observations o1 and o2 and two hypotheses h1 and h2, where we know that the se- quence of events was o1 → (h1|h2) → o2. The task then is to decide which of the two hypotheses is the more plausible one. An example from Bhagavatula et al. (2020) is the following: o1 : It was lunchtime and Kat was hungry. o2 : Kat and her coworkers enjoyed a nice lunch outside. h1 : Kat went to get a salad. h2 : Kat decided to take a nap instead of eating. While it is not inconceivable that someone would decide to take a nap …

Content truncated to 500 words