LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1235–1241 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1235 LEDGAR: A Large-Scale Multilabel Corpus for Text Classification of Legal Provisions in Contracts Don Tuggener, Pius von D¨aniken, Thomas Peetz, Mark Cieliebak Zurich University of Applied Sciences, Winterthur, Switzerland Legartis technology AG, Zurich, Switzerland {tuge, vode, ciel}@zhaw.ch, thomas.peetz@legartis.ai Abstract We present LEDGAR, a multilabel corpus of legal provisions in contracts. The corpus was crawled and scraped from the public domain (SEC filings) and is, to the best of our knowledge, the first freely available corpus of its kind. Since the corpus was constructed semi-automatically, we apply and discuss various approaches to noise removal. Due to the rather large labelset of over 12,000 labels annotated in almost 100,000 provisions in over 60,000 contracts, we believe the corpus to be of interest for research in the field of Legal NLP, (large-scale or extreme) text classification, as well as for legal studies. We discuss several methods to sample subcopora from the corpus and implement and evaluate different automatic classification approaches. Finally, we perform transfer experiments to evaluate how well the classifiers perform on contracts stemming from outside the corpus. Keywords: multilabel text classification, legal nlp, corpus creation 1. Introduction Legal Natural Language Processing is an emerging field that investigates the application of Natural Language Pro- cessing (NLP) techniques to the legal domain. Several ded- icated conferences and workshops have emerged in recent years (Aletras et al., 2019; Rehm et al., 2018; Palmirani, 2018; Keppens and Governatori, 2017). As a relatively young discipline, Legal NLP lacks resources in several ar- eas. We aim to address this lack regarding text classifica- tion by presenting LEDGAR (Labeled EDGAR1), a corpus of labeled provisions in contracts.2 Contractual provisions are a primary research target in law studies (Mills, 2019; Knapp et al., 2019; Hershkoff and Ka- han, 2018; Fisch, 2018, inter alia), as they are the essen- tial discourse units when drafting, negotiating, validating, or analysing contracts. Each provision in a contract con- stitutes a legal speech act (Yovel, 2000; Trosborg, 1995; Trosborg, 1991), and the concatenation of the provisions comprises the legal essence of a contract. Meanwhile, an emerging line of research investigates text classification with large-scale or extreme labelsets, where labelsets consist of thousands or even millions of labels (Bengio et al., 2019; Choromanska and Kumar Jain, 2019; Soni et al., 2018, e.g.). In this light, our main contributions are: • We provide a freely available, substantially-sized cor- pus (100m+ tokens) that assists the study of provi- sions, both from a legal and an NLP perspective. • The corpus features a large labelset (12k+ labels), which makes it attractive to research in the field of large-scale (or extreme) multilabel and multiclass classification. 1Cf. Section 3.1. for the explanation of the acronym. 2Available here: https://drive.switch.ch/index. php/s/j9S0GRMAbGZKa1A • Additionally, we discuss several subsampling tech- niques and present a method for extracting a label hier- archy based on label names. This …

Content truncated to 500 words