An Evaluation Framework for Legal Document Summarization

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4747–4753 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4747 An Evaluation Framework for Legal Document Summarization Ankan Mullick♠† Abhilash Nandy♠♢† Manav Nitin Kapadnis♠† Sohan Patnaik♠ R Raghav♠ Roshni Kar♠ ♠Indian Institute of Technology Kharagpur ♢ L3S Research Center, Leibniz Universit¨at Hannover {ankanm, nandyabhilash}@kgpian.iitkgp.ac.in, {iammanavk, sohanpatnaik106, rraghav5600, roshnikar}@iitkgp.ac.in Abstract A law practitioner has to go through numerous lengthy legal case proceedings for their practices of various categories, such as land dispute, corruption, etc. Hence, it is important to summarize these documents, and ensure that summaries contain phrases with intent matching the category of the case. To the best of our knowledge, there is no evaluation metric that evaluates a summary based on its intent. We propose an automated intent-based summarization metric, which shows a better agreement with human evaluation as compared to other automated metrics like BLEU, ROUGE-L etc. in terms of human satisfaction. We also curate a dataset by annotating intent phrases in legal documents, and show a proof of concept as to how this system can be automated. Additionally, all the code and data to generate reproducible results is available on Github. Keywords: Summarization, Evaluation Methodologies, Information Extraction, Legal Dataset 1. Introduction Summarization could be extractive, where the sum- mary has spans from the original text or abstractive, where the summary is generated using the original text. (Sai et al., 2020), (Chen et al., 2019a) list various metrics to evaluate summarization. BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE (Lin, 2004) etc. are context-free metrics, which work well for extractive summarization, while SMS (Clark et al., 2019), BERTScore (Zhang* et al., 2020) etc. are context-based metrics, which work well for abstractive summarization. Automatic summariza- tion of legal documents (Bhattacharya et al., 2019) is required because - (1) Average length of an any Court Judgement is as high as 4, 500 words (for example - Indian Supreme Court Judgments) (2) A law practi- tioner has to go through all contents of previous le- gal proceedings manually (3) Hiring Legal experts to summarize legal documents is expensive and very time consuming. However, while evaluating the quality of summaries, existing metrics fail when evaluating the amount of intent in the original text that is captured by the summary (intent here refers to the intention la- tent in a piece of text. e.g. (a)‘Accused No. 1 Balwan Singh (appellant in Criminal Appeal No. 727 of 2015), on 22nd January, 2007, at evening time, was talking with the other accused regarding preparation to kill’ - in this sentence, the phrase ‘preparation to kill’ depicts the intent of Murder (b) ‘In the case in hand, robbed ar- ticles were found to be kept concealed at a place within knowledge of the applicant/accused No.1, and there- fore, he is presumed to be one of the decoit involved in the decoity at the house of the first Aarti Palkar’ - in this sentence, the phrases …

Content truncated to 500 words