FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4389 - 4406 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing Ilias Chalkidis †∗ Tommaso Pasini † Sheng Zhang ⋄ Letizia Tomada ‡ Sebastian Felix Schwemer ‡ Anders Søgaard † † Department of Computer Science, University of Copenhagen, Denmark ‡ Faculty of Law, University of Copenhagen, Denmark ⋄ National University of Defense Technology, People’s Republic of China Abstract We present a benchmark suite of four datasets for evaluating the fairness of pre-trained lan- guage models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (Euro- pean Council, USA, Switzerland, and China), five languages (English, German, French, Ital- ian and Chinese) and fairness across five at- tributes (gender, age, region, language, and le- gal area). In our experiments, we evaluate pre- trained language models using several group- robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our re- sults, highlighting open challenges in the de- velopment of robustness methods in legal NLP. 1 Introduction Natural Language Processing (NLP) for law (Chalkidis and Kampas, 2019; Aletras et al., 2019; Zhong et al., 2020; Chalkidis et al., 2022) receives increasing attention. Assistive technologies can speed up legal research or discovery significantly assisting lawyers, judges and clerks. They can also help legal scholars to study case law (Katz, 2012; Coupette et al., 2021), improve access of law to laypersons, help sociologists and research ethicists to expose biases in the justice system (Angwin et al., 2016; Dressel and Farid, 2018), and even scrutinize decision-making itself (Bell et al., 2021). In the context of law, the principle of equal- ity and non-discrimination is of paramount impor- tance, although its definition varies at international, regional and domestic level. For example, EU non- discrimination law prohibits both direct and indi- rect discrimination. Direct discrimination occurs when one person is treated less favourably than ∗ Corresponding author: ilias.chalkidis@di.ku.dk Figure 1: Group disparity for defendant state (C.E. Eu- rope vs. The Rest) in ECtHR and legal area (Civil law vs. Penal law) in FSCS. others would be treated in comparable situations on grounds of sex, racial or ethnic origin, disabil- ity, sexual orientation, religion or belief and age.1 Given the gravity that legal outcomes have for indi- viduals, assistive technologies cannot be adopted to speed up legal research at the expense of fairness (Wachter et al., 2021), potentially also decreasing the trust in our legal systems (Barfield, 2020). Societal transformations perpetually shape our legal systems. The topic deserves great attention because AI systems learning from historical data pose the risk of lack of generalisability beyond the training data, and more importantly transporting biases previously encumbered in the data in future decision-making, thereby exponentially increasing their effect (Delacroix, 2022). Historical legal …

Content truncated to 500 words