Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4254–4268 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4254 Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines Łukasz Borchmann and Dawid Wi´sniewski and Andrzej Gretkowski Izabela Kosmala and Dawid Jurkiewicz and Łukasz Szałkiewicz Gabriela Pałka and Karol Kaczmarek and Agnieszka Kaliska and Filip Grali´nski Applica.ai, Warsaw, Poland {firstname.surname}@applica.ai Abstract We propose a new shared task of semantic re- trieval from legal texts, in which a so-called contract discovery is to be performed–where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. The task differs substantially from conventional NLI and shared tasks on legal information extraction (e.g., one has to identify text span instead of a single document, page, or paragraph). The specification of the proposed task is followed by an evaluation of multiple solutions within the unified frame- work proposed for this branch of methods. It is shown that state-of-the-art pretrained encoders fail to provide satisfactory results on the task proposed. In contrast, Language Model-based solutions perform better, especially when un- supervised fine-tuning is applied. Besides the ablation studies, we addressed questions re- garding detection accuracy for relevant text fragments depending on the number of exam- ples available. In addition to the dataset and reference results, LMs specialized in the legal domain were made publicly available. 1 Introduction Processing of legal contracts requires significant human resources due to the complexity of docu- ments, the expertise required and the consequences at stake. Therefore, a lot of effort has been made to automate such tasks in order to limit process- ing costs–notice that law was one of the first ar- eas where electronic information retrieval systems were adopted (Maxwell and Schafer, 2008). Enterprise solutions referred to as contract dis- covery deal with tasks, such as ensuring the in- clusion of relevant clauses or their retrieval for further analysis (e.g., risk assessment). Such pro- cesses can consist of a manual definition of a few examples, followed by conventional information Task Legal SI Few-shot COLIEE + − − SNLI − − − MultiNLI − − − TREC Legal Track + − − Propaganda detection − + − THUMOS (video) − + + ActivityNet (video) − + + ALBAYZIN (audio) − + − Contract Discovery (ours) + + + Table 1: Comparison of existing shared tasks. Most of the related NLP tasks do not assume Span Identifica- tion (SI), even those outside the legal domain (Legal). Moreover, the few-shot setting is not popular within the field of NLP yet. retrieval. This approach was taken recently by Nag- pal et al. (2018) for the extraction of fairness poli- cies spread across agreements and administrative regulations. 2 Review of Existing Datasets Table 1 summarizes main differences between available challenges. It is shown that most of the re- lated NLP tasks do not assume span identification, even those outside the legal domain. Moreover, the few-shot setting is not popular within the field of …

Content truncated to 500 words