Neural Legal Judgment Prediction in English

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4317 Neural Legal Judgment Prediction in English Ilias Chalkidis∗ Ion Androutsopoulos∗ Nikolaos Aletras∗∗ ∗ Department of Informatics, Athens University of Economics and Business, Greece ∗∗ Computer Science Department, University of Sheffield, UK [ihalk,ion]@aueb.gr, n.aletras@sheffield.ac.uk Abstract Legal judgment prediction is the task of au- tomatically predicting the outcome of a court case, given a text describing the case’s facts. Previous work on using neural models for this task has focused on Chinese; only feature- based models (e.g., using bags of words and topics) have been considered in English. We release a new English legal judgment predic- tion dataset, containing cases from the Euro- pean Court of Human Rights. We evaluate a broad variety of neural models on the new dataset, establishing strong baselines that sur- pass previous feature-based models in three tasks: (1) binary violation classification; (2) multi-label classification; (3) case importance prediction. We also explore if models are biased towards demographic information via data anonymization. As a side-product, we propose a hierarchical version of BERT, which bypasses BERT’s length limitation. 1 Introduction Legal information is often represented in textual form (e.g., legal cases, contracts, bills). Hence, le- gal text processing is a growing area in NLP with various applications such as legal topic classifi- cation (Nallapati and Manning, 2008; Chalkidis et al., 2019), court opinion generation (Ye et al., 2018) and analysis (Wang et al., 2012), legal infor- mation extraction (Chalkidis et al., 2018), and en- tity recognition (Cardellino et al., 2017; Chalkidis et al., 2017). Here, we focus on legal judgment prediction, where given a text describing the facts of a legal case, the goal is to predict the court’s out- come (Aletras et al., 2016; S¸ulea et al., 2017; Luo et al., 2017; Zhong et al., 2018; Hu et al., 2018). Such models may assist legal practitioners and citizens, while reducing legal costs and improv- ing access to justice (Lawlor, 1963; Katz, 2012; Stevenson and Wagoner, 2015). Lawyers and judges can use them to estimate the likelihood of winning a case and come to more consistent and informed judgments, respectively. Human rights organizations and legal scholars can employ them to scrutinize the fairness of judicial decisions un- veiling if they correlate with biases (Doshi-Velez and Kim, 2017; Binns et al., 2018). This paper contributes a new publicly avail- able English legal judgment prediction dataset of cases from the European Court of Human Rights (ECHR).1 Unlike Aletras et al. (2016), who pro- vide only features from approx. 600 ECHR cases, our dataset is substantially larger (∼11.5k cases) and provides access to the raw text. As a sec- ond contribution, we evaluate several neural mod- els in legal judgment prediction for the first time in English. We consider three tasks: (1) binary classification (i.e., violation of a human rights ar- ticle or not), the only task considered by Ale- tras et al. (2016); …

Content truncated to 500 words