DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4913–4922 August 1–6, 2021. ©2021 Association for Computational Linguistics 4913 DOCNLI: A Large-scale Dataset for Document-level Natural Language Inference Wenpeng Yin1, Dragomir Radev1,2 and Caiming Xiong1 1Salesforce Research and 2Yale University wyin@salesforce.com Abstract Natural language inference (NLI) is formu- lated as a unified framework for solving vari- ous NLP problems such as relation extraction, question answering, summarization, etc. It has been studied intensively in the past few years thanks to the availability of large-scale labeled datasets. However, most existing studies fo- cus on merely sentence-level inference, which limits the scope of NLI’s application in down- stream NLP problems. This work presents DOCNLI — a newly-constructed large-scale dataset for document-level NLI. DOCNLI is transformed from a broad range of NLP prob- lems and covers multiple genres of text. The premises always stay in the document granu- larity, whereas the hypotheses vary in length from single sentences to passages with hun- dreds of words. Additionally, DOCNLI has pretty limited artifacts1 which unfortunately widely exist in some popular sentence-level NLI datasets. Our experiments demonstrate that, even without fine-tuning, a model pre- trained on DOCNLI shows promising perfor- mance on popular sentence-level benchmarks, and generalizes well to out-of-domain NLP tasks that rely on inference at document gran- ularity. Task-specific fine-tuning can bring fur- ther improvements. Data, code and pretrained models can be found at https://github. com/salesforce/DocNLI. 1 Introduction A fundamental challenge of natural language pro- cessing (NLP) lies in the variability of semantic expression, where the same meaning can be con- veyed by, or inferred from, different pieces of text (Dagan et al., 2009). This phenomenon gives rise to the many-to-many mapping between textual ex- pressions and meanings. Many NLP problems, 1NLI “artifacts” are some label-specific biases (often) in the hypotheses; they can indicate which NLI class a hypothesis belongs to even without looking at the premise. such as information extraction, question answering, document summarization and machine translation, desire a system for this variability phenomenon so as to figure out that a particular meaning can be inferred from distinct text strings (Dagan et al., 2009). Natural language inference (a.k.a textual entailment (Dagan et al., 2005)) acts as a unified framework to study those NLP problems by cast- ing the background text as a premise and the text of target meaning as a hypothesis. Then, a good NLI recognizer can be considerably translated to a well-performing system regarding respective NLP tasks. NLI was first studied in (Dagan et al., 2005). Research in the early stages was mostly driven by the PASCAL Recognizing Textual Entailment (RTE) challenges which are annual competitions with benchmark datasets released. In the past few years, the study of NLI has moved forward rapidly along with the construction of large-scale datasets, such as SNLI (Bowman et al., 2015), the science domain SciTail (Khot et al., 2018) and multi-genre MNLI (Williams et al., 2018), etc. However, some NLI datasets may not be suitable any more for solving downstream NLP problems since they were …

Content truncated to 500 words