LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4310 - 4330 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LexGLUE: A Benchmark Dataset for Legal Language Understanding in English Ilias Chalkidis α∗ Abhik Jana β Dirk Hartung γ δ Michael Bommarito γ δ Ion Androutsopoulos ϵ Daniel Martin Katz γ δ ζ Nikolaos Aletras η α University of Copenhagen, Denmark β Universit¨at Hamburg, Germany γ Bucerius Law School, Hamburg, Germany δ CodeX, Stanford Law School, United States ϵ Athens University of Economics and Business, Greece η University of Sheffield, UK ζ Illinois Tech – Chicago Kent College of Law, United States Abstract Laws and their interpretations, legal arguments and agreements are typically expressed in writ- ing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) tech- nologies can be a valuable tool to support le- gal practitioners in these endeavors. Their use- fulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we intro- duce the Legal General Language Understand- ing Evaluation (LexGLUE) benchmark, a col- lection of datasets for evaluating model perfor- mance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improve- ments across multiple tasks. 1 Introduction Law is a field of human endeavor dominated by the use of language. As part of their professional training, law students consume large bodies of text as they seek to tune their understanding of the law and its application to help manage human behavior. Virtually every modern legal system produces massive volumes of textual data (Katz et al., 2020). Lawyers, judges, and regulators con- tinuously author legal documents such as briefs, memos, statutes, regulations, contracts, patents and judicial decisions (Coupette et al., 2021). Beyond the consumption and production of language, law and the art of lawyering is also an exercise centered around the analysis and interpretation of text. Natural language understanding (NLU) technolo- gies can assist legal practitioners in a variety of legal tasks (Chalkidis and Kampas, 2018; Aletras ∗ Corresponding author: ilias.chalkidis@di.ku.dk Figure 1: LexGLUE: A new benchmark dataset to eval- uate the capabilities of NLU models on legal text. et al., 2019, 2020; Zhong et al., 2020b; Bommarito et al., 2021), from judgment prediction (Aletras et al., 2016; Sim et al., 2016; Katz et al., 2017; Zhong et al., 2018; Chalkidis et al., 2019a; Malik et al., 2021), information extraction from legal doc- uments (Chalkidis et al., 2018, 2019c; Chen et al., 2020; Hendrycks et al., 2021) and case summariza- tion (Bhattacharya et al., 2019) to legal question an- swering (Ravichander et al., 2019; Kien et al., 2020; Zhong et al., 2020a,c) and text classification (Nal- lapati and …

Content truncated to 500 words