Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1048–1064 November 20–23, 2022. ©2022 Association for Computational Linguistics 1048 Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation Abhay Shukla1 Paheli Bhattacharya1 Soham Poddar1 Rajdeep Mukherjee1 Kripabandhu Ghosh2 Pawan Goyal1 Saptarshi Ghosh1∗ 1Indian Institute of Technology Kharagpur, India 2Indian Institute of Science Education and Research Kolkata, India Abstract Summarization of legal case judgement docu- ments is a challenging problem in Legal NLP. However, not much analyses exist on how dif- ferent families of summarization models (e.g., extractive vs. abstractive) perform when ap- plied to legal case documents. This question is particularly important since many recent transformer-based abstractive summarization models have restrictions on the number of in- put tokens, and legal documents are known to be very long. Also, it is an open question on how best to evaluate legal case document sum- marization systems. In this paper, we carry out extensive experiments with several extractive and abstractive summarization methods (both supervised and unsupervised) over three legal summarization datasets that we have developed. Our analyses, that includes evaluation by law practitioners, lead to several interesting insights on legal summarization in specific and long document summarization in general. 1 Introduction In Common Law systems (followed in India, UK, USA, etc.) law practitioners have to read through hundreds of case judgements/rulings in order to identify relevant cases that they can cite as prece- dents in an ongoing case. This is a time-consuming process as case documents are generally very long and complex. Thus, automatic summarization of legal case documents is an important problem (Gel- bart and Smith, 1991; Bhattacharya et al., 2019; Zhong et al., 2019; Liu and Chen, 2019). It is ad- ditionally challenging due to two primary reasons as demonstrated in Table 1 – (i) legal documents as well as their summaries are much longer than most other types of documents, and (ii) since it is expensive to get Law Experts to write summaries, the datasets are usually much smaller, making it difficult to use supervised models. ∗Corresponding author: saptarshi@cse.iitkgp.ac.in Dataset Language Domain #Doc Avg # Tokens Doc Summ CNN/DM (Hermann et al., 2015) EN News 312K 781 56 Gigawords (Napoles et al., 2012) EN News 4.02M 31 8 arXiv (Cohan et al.) EN Academic 216K 6,914 293 PubMed (Cohan et al.) EN Academic 133K 3,224 214 TL;DR, TOS;DR (Manor and Li, 2019) EN Contracts 506 106 17 BigPatent (Sharma et al.) EN Patent 1.34M 3,573 117 RulingBR (Feijó and Moreira, 2018) Portugese Court Rulings 10,623 1,397 100 This work IN-Ext (Indian docs, extractive summ) EN Court Rulings 50 5,389 1,670 IN-Abs (Indian docs, abstractive summ) EN Court Rulings 7,130 4,378 1,051 UK-Abs (UK docs, abstractive summ) EN Court Rulings 793 14,296 1,573 Table 1: Comparing some existing summarization datasets with the three legal summarization datasets developed in this work. Last two columns give the aver- age number …

Content truncated to 500 words