Exploiting Domain-Specific Knowledge for Judgment Prediction Is No Panacea

Proceedings of Recent Advances in Natural Language Processing, pages 1234–1243 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_139 1234 Exploiting Domain-Specific Knowledge for Judgment Prediction is no Panacea Olivier Sala¨un1, Philippe Langlais1, and Karim Benyekhlef2 1RALI, DIRO, University of Montr´eal {salaunol,felipe}@iro.umontreal.ca 2Cyberjustice Laboratory, Faculty of Law, University of Montr´eal karim.benyekhlef@umontreal.ca Abstract Legal judgment prediction (LJP) usually con- sists in a text classification task aimed at pre- dicting the verdict on the basis of the fact de- scription. The literature shows that the use of articles as input features helps improve the classification performance. In this work, we designed a verdict prediction task based on landlord-tenant disputes and we applied BERT- based models to which we fed different article- based features. Although the results obtained are consistent with the literature, the improve- ments with the articles are mostly obtained with the most frequent labels, suggesting that pre-trained and fine-tuned transformer-based models are not scalable as is for legal reason- ing in real life scenarios as they would only excel in accurately predicting the most recur- rent verdicts to the detriment of other legal out- comes. 1 Introduction At the intersection of machine learning and law, legal judgment prediction (LJP) is a task that con- sists in predicting either the outcome of a lawsuit (Skalak, 1989; Nallapati and Manning, 2008; Katz et al., 2017; Aletras et al., 2016; Liu and Chen, 2017) or some other case attributes such as legal ar- eas (S¸ulea et al., 2017; Soh et al., 2019) or charges (Xiao et al., 2018). One specificity of court rulings is that they are based on the application of legal articles to the fac- tual description of the case. That is to say, a judge must determine whether some law articles are rele- vant to a case, and if applicable, whether the legal principles they embody are violated. Therefore, ar- ticles as domain-specific knowledge can be used as leverage for improving LJP performance, as shown by Luo et al. (2017) and Long et al. (2019) for charge prediction and divorce judgment prediction respectively. Xu et al. (2020) also went further by using articles for distinguishing confusing charges in a charge prediction task. Meanwhile, transformers (Vaswani et al., 2017) and BERT models (Devlin et al., 2019) in partic- ular have been widely used in NLP tasks with the assumption that such models, if first pre-trained on massive corpora and then fine-tuned on the dataset of a given task, could suffice for achieving signifi- cant improvements. On one hand, this turned out to be true with the CAIL2018 dataset (charge pre- diction task) as shown by Wang et al. (2020). On the other hand, Holzenberger et al. (2020) men- tioned in a statutory reasoning entailment task that a transformer model does worse than a rule-based model, even after further pre-training on the do- main corpus. Furthermore, in an employment no- tice prediction task, Lam et al. (2020) emphasized that domain adaption of such models could even harm performance. These elements raise the ques- tion of how well …

Content truncated to 500 words