SciNLI: A Corpus for Natural Language Inference on Scientific Text

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7399 - 7409 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SCINLI: A Corpus for Natural Language Inference on Scientific Text Mobashir Sadat and Cornelia Caragea Computer Science University of Illinois Chicago msadat3@uic.edu cornelia@uic.edu Abstract Existing Natural Language Inference (NLI) datasets, while being instrumental in the ad- vancement of Natural Language Understand- ing (NLU) research, are not related to scien- tific text. In this paper, we introduce SCINLI, a large dataset for NLI that captures the for- mality in scientific text and contains 107, 412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. Given that the text used in scientific literature dif- fers vastly from the text used in everyday lan- guage both in terms of vocabulary and sen- tence structure, our dataset is well suited to serve as a benchmark for the evaluation of sci- entific NLU models. Our experiments show that SCINLI is harder to classify than the exist- ing NLI datasets. Our best performing model with XLNet achieves a Macro F1 score of only 78.18% and an accuracy of 78.23% showing that there is substantial room for improvement. 1 Introduction Natural Language Inference (NLI) or Textual En- tailment (Bowman et al., 2015) aims at recogniz- ing the semantic relationship between a pair of sentences—whether the second sentence entails the first sentence, contradicts it, or they are seman- tically independent. NLI was introduced (Dagan, Glickman, and Magnini, 2005) to facilitate the eval- uation of Natural Language Understanding (NLU) that significantly impacts the performance of many NLP tasks such as text summarization, question answering, and commonsense reasoning. To date, several NLI datasets are made available (Bowman et al., 2015; Williams, Nangia, and Bow- man, 2018; Marelli et al., 2014; Dagan, Glickman, and Magnini, 2005). These datasets have not only been instrumental for developing and evaluating NLI models but also have been useful in advanc- ing many other NLP areas such as: representation learning (Conneau et al., 2017), transfer learning (Pruksachatkun et al., 2020) and multi-task learn- ing (Liu et al., 2019a). However, despite their usefulness, none of the existing NLI datasets is related to scientific text that is found in research articles. The vocabulary as well as the structure and formality used in sen- tences in scientific articles are very different from the sentences used in the everyday language. More- over, the scientific text captured in research papers brings additional challenges and complexities not only in terms of the language and its structure but also the inferences that exist in it which are not available in the existing NLI datasets. For example, a sentence can present the reasoning behind the conclusion made in the previous sentence, while other sentences indicate a contrast or entailment with the preceding sentence. These inferences are crucial for understanding, analyzing, and reason- ing over scientific work (Luukkonen, 1992; Kuhn, 2012; Hall, Jurafsky, and Manning, 2008). There- fore, ideally, the scientific language inference mod- els …

Content truncated to 500 words