LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15513–15535 July 9-14, 2023 ©2023 Association for Computational Linguistics LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development Ilias Chalkidis∗ Nicolas Garneau∗ Anders Søgaard Department of Computer Science, University of Copenhagen, Denmark C˘at˘alina Goant, ˘a Utrecht University School of Law, Netherlands Daniel Martin Katz Illinois Tech – Chicago Kent College of Law, IL, United States Abstract In this work, we conduct a detailed analy- sis on the performance of legal-oriented pre- trained language models (PLMs). We exam- ine the interplay between their original ob- jective, acquired knowledge, and legal lan- guage understanding capacities which we de- fine as the upstream, probing, and downstream performance, respectively. We consider not only the models’ size but also the pre-training corpora used as important dimensions in our study. To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs. We release two new le- gal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We find that probing performance strongly correlates with upstream performance in related legal topics. On the other hand, downstream performance is mainly driven by the model’s size and prior legal knowledge which can be estimated by upstream and prob- ing performance. Based on these findings, we can conclude that both dimensions are im- portant for those seeking the development of domain-specific PLMs. 1 Introduction Following closely the advances in the development of NLP technologies, the legal NLP literature is flourishing with the release of many new resources, including large legal corpora (Henderson* et al., 2022), datasets (Chalkidis et al., 2021a; Koreeda and Manning, 2021; Zheng et al., 2021; Chalkidis et al., 2022a; Habernal et al., 2022), and pre-trained legal-oriented language models (PLMs) (Chalkidis et al., 2020; Zheng et al., 2021; Xiao et al., 2021). Benchmark suites (Chalkidis et al., 2022a; Hwang et al., 2022; Niklaus et al., 2023) to evaluate the performance of PLMs in a more systematic way ∗ Equal contribution. have been also developed, showcasing the superi- ority of legal-oriented PLMs over generic ones on downstream legal NLP tasks. Despite this impressive progress, there is still not a thorough study on (a) how PLMs trained under different settings (pre-training corpora, size of the model) perform across different legal sub- corpora, and (b) what sort of knowledge such mod- els have acquired from pre-training, and (c) how important is domain (legal) specificity vs general (cross-domain) legal knowledge. Furthermore, of- ten times, legal NLP relies on datasets without drawing clear lines and comparisons between the various legal systems they may reflect. A legal sys- tem may be defined as a set of rules adopted and enforced at a given governance level, which may be national, regional or international (Friedman and Hayden, 2017), e.g., UK, EU, US, CoE, etc. We define the upstream evaluation as the task PLMs are explicitly designed to do: …

Content truncated to 500 words