EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 809 - 824 December 7-11, 2022 ©2022 Association for Computational Linguistics DocInfer: Document-level Natural Language Inference using Optimal Evidence Selection Puneet Mathur∗1, Riyaz Bhat2, Gautam Kunapuli3, Manish Shrivastava4, Dinesh Manocha1, and Maneesh Singh3 1University of Maryland, College Park, MD, USA 2IBM, Bengaluru, India 3Motive, San Francisco, USA 4IIIT, Hyderabad, India Abstract We present DocInfer - a novel, end-to-end Document-level Natural Language Inference model that builds a hierarchical document graph enriched through inter-sentence rela- tions (topical, entity-based, concept-based), performs paragraph pruning using the novel SubGraph Pooling layer, followed by optimal evidence selection based on REINFORCE al- gorithm to identify the most important context sentences for a given hypothesis. Our evidence selection mechanism allows it to transcend the input length limitation of modern BERT-like Transformer models while presenting the en- tire evidence together for inferential reasoning. We show this is an important property needed to reason on large documents where the evi- dence may be fragmented and located arbitrar- ily far from each other. Extensive experiments on popular corpora - DocNLI, ContractNLI, and ConTRoL datasets, and our new proposed dataset called CaseHoldNLI on the task of legal judicial reasoning, demonstrate significant per- formance gains of 8-12% over SOTA methods. Our ablation studies validate the impact of our model. Performance improvement of ∼ 3−6% on annotation-scarce downstream tasks of fact verification, multiple-choice QA, and contract clause retrieval demonstrates the usefulness of DocInfer beyond primary NLI tasks. 1 Introduction Natural Language Inference (NLI) is a fundamental textual reasoning task seeking to classify a pre- sented hypothesis as entailed by, contradictory to or neutral to a premise (Dagan et al., 2010). Prior NLI datasets and studies have focused on sentence-level inference where both the premises and hypotheses are single sentences (SNLI (Bow- man et al., 2015), MultiNLI(Williams et al., 2018), QNLI and WNLI(Wang et al., 2018)) Document- level NLI extends the reasoning of NLI beyond ∗*Corresponding Author:puneetm@umd.edu sentence granularity where the premises are in the document granularity, whereas the hypotheses can vary in length from single sentences to passages with hundreds of words(Yin et al., 2021). Document level NLI is an important problem for many tasks including verification of factual cor- rectness of document summaries, fact-checking assertions against articles, QA on long texts, legal compliance of contracts, etc. Even so, it challenges modern approaches due to the limited input bot- tleneck of modern Transformer models. Consider that the universally used BERT model (Devlin et al., 2018) can only encode 512 input sub-tokens due to its quadratic self-attention complexity. Conse- quently, evidence in the document premise relevant to the hypothesis can potentially be distributed in several textual spans located arbitrarily far away from each other in long documents, and may not be simultaneously available to draw inference. Recent approaches, notably SpanNLI (Koreeda and Manning, 2021), HESM (Hanselowski et al., 2018)) and others, have shown that chunking the premise into multiple document spans, scoring them, and aggregating the scores helps mitigate the limited input …

Content truncated to 500 words