Steno AI at SemEval-2023 Task 6: Rhetorical Role Labelling of Legal Documents using Transformers and Graph Neural Networks

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1858–1862 July 13-14, 2023 ©2023 Association for Computational Linguistics Steno AI at SemEval-2023 Task 6: Rhetorical Role Labeling of Legal Documents using Transformers and Graph Neural Networks Anshika Gupta, Shaz Furniturewala, Vijay Kumari, Yashvardhan Sharma BITS Pilani,Pilani,Rajasthan (f20200111,f20200025)@pilani.bits-pilani.ac.in (p20190065,yash)@pilani.bits-pilani.ac.in Abstract A legal document is usually long and dense requiring human effort to parse it. It also contains significant amounts of jargon which make deriving insights from it using existing models a poor approach. This paper presents the approaches undertaken to perform the task of rhetorical role labelling on Indian Court Judgements as part of SemEval Task 6: understanding legal texts, shared subtask A (Modi et al., 2023). We experiment with graph based approaches like Graph Convolutional Networks and Label Propagation Algorithm, and transformer-based approaches including variants of BERT to improve accuracy scores on text classification of complex legal documents. 1 Introduction Rhetorical Role Labelling for Legal Documents refers to the task of classifying sentences from court judgements into various categories depending on their semantic function in the document. This task is important as it not only has direct applica- tions in the legal industry but also has the ability to aid several other tasks on legal documents such as summarization and legal search. This task is still in it’s early stages, with huge scope for improvement over the current state-of-the-art. To facilitate automatic interpretation of legal doc- uments by dividing them into topic coherent com- ponents, a rhetorical role corpus was created for Task 6, sub-task A of The International Workshop on Semantic Evaluation (Modi et al., 2023). Sev- eral applications of legal AI, including judgment summarizing, judgment outcome prediction, prece- dent search, etc., depend on this classification. 2 Related Works with Comparison The predominant technique used in Rhetorical Role Labeling over large datasets is based on the use Model F1 score LEGAL-BERT 0.557 LEGAL-BERT + Neural Net 0.517 ERNIE 2.0 0.505 Table 1: Summary of related works on the task of rhetor- ical role labelling on legal text. (Parikh et al., 2022) of transformer-based models like LEGAL-BERT (Chalkidis et al., 2020) and ERNIE 2.0 (Sun et al., 2020), augmented by various heuristics or neural network models. The accuracy of these approaches has remained low over the years. The results are summarized in Table 1. The dataset (Parikh et al., 2022) used to imple- ment the above approaches is relatively small, con- sisting only of a few hundred annotated documents and 7 sentence classes. 3 Dataset The dataset (Kalamkar et al., 2022) is made up of publicly available Indian Supreme Court Judge- ments. It consists of 244 train documents, 30 vali- dation documents and 50 test documents making a total of 36023 sentences. For every document, each sentence has been catego- rized into one of 13 semantic categories as follows: 1. PREAMBLE: The initial sentences of a judgement mentioning the relevant parties 2. FAC: Sentences that describe the events that led to the filing of the case 3. …

Content truncated to 500 words