ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4046–4062 August 1–6, 2021. ©2021 Association for Computational Linguistics 4046 ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation Vijit Malik1 Rishabh Sanjay1 Shubham Kumar Nigam1 Kripa Ghosh2 Shouvik Kumar Guha3 Arnab Bhattacharya1 Ashutosh Modi1 1Indian Institute of Technology Kanpur (IIT-K) 2Indian Institute of Science Education and Research Kolkata (IISER-K) 3West Bengal National University of Juridical Sciences (WBNUJS) {vijitvm,rsan,sknigam}@iitk.ac.in kripaghosh@iiserkol.ac.in shouvikkumarguha@nujs.edu {arnabb,ashutoshm}@cse.iitk.ac.in Abstract An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a sys- tem to be practically useful, predictions by the system should be explainable. To promote re- search in developing such a system, we intro- duce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court de- cisions. A portion of the corpus (a separate test set) is annotated with gold standard ex- planations by legal experts. Based on ILDC, we propose the task of Court Judgment Pre- diction and Explanation (CJPE). The task re- quires an automated system to predict an ex- plainable outcome of a case. We experiment with a battery of baseline models for case pre- dictions and propose a hierarchical occlusion based model for explainability. Our best pre- diction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed al- gorithm reveals a significant difference in the point of view of the algorithm and legal ex- perts for explaining the judgments, pointing to- wards scope for future research. 1 Introduction In many of the highly populated countries like In- dia, there is a vast number of pending backlog of legal cases that impede the judicial process (Katju, 2019). The backlog is due to multiple factors, including the unavailability of competent judges. Therefore, a system capable of assisting a judge by suggesting the outcome of an ongoing court case is likely to be useful for expediting the judicial process. However, an automated decision system is not tenable in law unless it is well explained in terms of how humans understand the legal process. Hence, it is necessary to explain the suggestion. In other words, we would like such a system to pre- dict not only what should be the final decision of a court case but also how one arrives at that decision. In this paper, we introduce INDIAN LEGAL DOC- UMENTS CORPUS (ILDC) intending to promote research in developing a system that could assist in legal case judgment prediction in an explainable way. ILDC is a corpus of case proceedings from the Supreme Court of India (SCI) that are annotated with original court decisions. A portion of ILDC (i.e., a separate test set) is additionally annotated with gold standard judgment decision explanations by legal experts to …

Content truncated to 500 words