AntContentTech at SemEval-2023 Task 6: Domain-adaptive Pretraining and Auxiliary-task Learning for Understanding Indian Legal Texts

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 402–408 July 13-14, 2023 ©2023 Association for Computational Linguistics AntContentTech at SemEval-2023 Task 6: Domain-adaptive Pretraining and Auxiliary-task Learning for Understanding Indian Legal Texts Jingjing Huo, Kezun Zhang, Zhengyong Liu, Xuan Lin∗, Wenqiang Xu Maozong Zheng, Zhaoguo Wang, Song Li FinContentTech, Ant Group {huojingjing.hjj, kezun.zkz, liuzhengyong.lzy, daxuan.lx, yugong.xwq, zhengmaozong.zmz, wangzhaoguo.wzg, hejian.ls}@antgroup.com Abstract The objective of this shared task is to gain an understanding of legal texts, and it is be- set with difficulties such as the comprehen- sion of lengthy noisy legal documents, domain specificity as well as the scarcity of annotated data. To address these challenges, we propose a system that employs a hierarchical model and integrates domain-adaptive pretraining, data augmentation, and auxiliary-task learning tech- niques. Moreover, to enhance generalization and robustness, we ensemble the models that utilize these diverse techniques. Our system ranked first on the RR sub-task and in the mid- dle for the other two sub-tasks. Our code is publicly available here1. 1 Introduction Certain heavily populated nations, such as India, are faced with a surfeit of unresolved legal cases, and the development of automated AI tools can potentially aid legal practitioners in expediting the judicial system. Due to the intricate nature of the judicial process, the SemEval-2023 Task 6 (Modi et al., 2023) proposes three sub-tasks that can serve as foundational components for creating legal AI applications, i.e. Sub-task A: Rhetorical Roles Pre- diction (RR), Sub-task B: Legal Named Entities Extraction (L-NER) and Sub-task C: Court Judge- ment Prediction with Explanation (CJPE). The RR sub-task is to organize unstructured legal documents into semantically cohesive units (clas- sification at the sentence level), while the second L-NER sub-task involves identifying relevant en- tities within a legal document. The third CJPE sub-task focuses on predicting the outcome of a judgment (classification at the document level) and providing a corresponding explanation. ∗Corresponding author 1https://github.com/ContentTech/ rhetorical-role-baseline The task at hand focusing on Indian legal docu- ments is inherently challenging. Firstly, legal doc- uments exhibit considerably greater length, and a higher degree of unstructured content and noise when compared to shorter documents used for the training of traditional natural language processing models. Moreover, the presence of legal-specific lexicons within the documents makes general pre- trained neural models ineffective. Further, legal provisions vary from one nation to another, and the legal domain also has several sub-domains that align with different legal provisions. It follows that a pretrained model equipped with legal knowledge specific to one sub-domain may not be applicable to another distinct sub-domain. Another key chal- lenge in this task is the limited size of annotated data due to the time-consuming and expensive an- notation process (Kalamkar et al., 2022). To address the aforementioned challenges, we propose a system based on the baseline model called SciBERT-HSLN in (Kalamkar et al., 2022; Brack et al., 2021) and equipped with domain- adaptive pretraining, data augmentation strategies, as well as auxiliary-task learning techniques. In summary, we list here our findings that are …

Content truncated to 500 words