IRIT_IRIS_C at SemEval-2023 Task 6: A Multi-level Encoder-based Architecture for Judgement Prediction of Legal Cases and their Explanation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 686–692 July 13-14, 2023 ©2023 Association for Computational Linguistics IRIT_IRIS_C at SemEval-2023 Task 6: A Multi-level Encoder-based Architecture for Judgement Prediction of Legal Cases and their Explanation Nishchal Prasad, Mohand Boughanem, Taoufiq Dkaki Institut de Recherche en Informatique de Toulouse (IRIT), Toulouse, France {Nishchal.Prasad, Mohand.Boughanem, Taoufiq.Dkaki}@irit.fr Abstract This paper describes our system used for sub- task C (1 & 2) in Task 6: LegalEval: Under- standing Legal Texts. We propose a three-level encoder-based classification architecture that works by fine-tuning a BERT-based pre-trained encoder, and post-processing the embeddings extracted from its last layers, using transformer encoder layers and RNNs. We run ablation studies on the same and analyze its perfor- mance. To extract the explanations for the pre- dicted class we develop an explanation extrac- tion algorithm, exploiting the idea of a model’s occlusion sensitivity. We explored some train- ing strategies with a detailed analysis of the dataset. Our system ranked 2nd (macro-F1 met- ric) for its sub-task C-1 and 7th (ROUGE-2 metric) for sub-task C-2. 1 Introduction With the increasing population, there is also an increase in the number of legal cases, and the short- age of capable judges to attend to them makes it more difficult to process them in time, hence cre- ating backlogs. In such a scenario, a method to assist the judges and suggest an outcome (with an explanation) of a legal case becomes necessary. Such methods can also assist a legal professional to help people; unfamiliar with the legal terms and proceedings; to make informed decisions. In the past, there have been many such approaches us- ing machine learning techniques. In Sub-task C of Task 6: LegalEval (Modi et al. (2023)) the or- ganizers tried to tackle this problem in the context of Indian Legal Cases from the Supreme Court of India (SCI). Task C is divided into two sub-tasks, C-1 and C-2. C-1 is the judgment prediction of the court cases/documents, and C-2 is the explanation of the said prediction. In this work, we propose a classification frame- work taking a few ideas from the hierarchical trans- former approach by Pappagari et al. (2019) which builds upon BERT (Devlin et al. (2019)) as a back- bone encoder model. We fine-tune a BERT-based pre-trained model and experiment with features extracted from its last layers, and use it for fur- ther classification (sub-task C-1) process, through a combination of transformer encoders and RNNs. We explored the effect of training this framework with different parts of the training dataset and pro- vide an analysis of its impact. For the explanation task (sub-task C-2) we exploit the idea of “occlu- sion sensitivity” (Zeiler and Fergus (2014)) and de- velop a mechanism to extract the relevant sentences from a document which serves as the explanation for its prediction using the classification method in sub-task C-1. Our model ranked 2nd1 out of 11 participating teams on sub-task C-1 obtaining 0.7228 macro-F1, and 7th1 out of 11 on sub-task …

Content truncated to 500 words