SemEval-2023 Task 6: LegalEval - Understanding Legal Texts

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2362–2374 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 6: LegalEval - Understanding Legal Texts Ashutosh Modi¶ Prathamesh Kalamkar† Saurabh Karn‡ Aman Tiwari† Abhinav Joshi¶ Sai Kiran Tanikella¶ Shouvik Kumar Guha⋄ Sachin Malhan‡ Vivek Raghavan⋆ ¶Indian Institute of Technology Kanpur (IIT-K), †Thoughtworks, ‡Agami, ⋆EkStep, ⋄NUJS, West Bengal {ashutoshm, ajoshi, tskiran}@cse.iitk.ac.in, {prathamk, aman.tiwari}@thoughtworks.com, {saurabh, sachin}@agami.in, shouvikkumarguha@nujs.edu, vivek@ekstep.org Abstract In populous countries, pending legal cases have been growing exponentially. There is a need for developing NLP-based techniques for pro- cessing and automatically understanding legal documents. To promote research in the area of Legal NLP we organized the shared task Legal- Eval - Understanding Legal Texts at SemEval 2023. LegalEval task has three sub-tasks: Task- A (Rhetorical Roles Labeling) is about automat- ically structuring legal documents into seman- tically coherent units, Task-B (Legal Named Entity Recognition) deals with identifying rel- evant entities in a legal document and Task-C (Court Judgement Prediction with Explanation) explores the possibility of automatically pre- dicting the outcome of a legal case along with providing an explanation for the prediction. In total 26 teams (approx. 100 participants spread across the world) submitted systems paper. In each of the sub-tasks, the proposed systems out- performed the baselines; however, there is a lot of scope for improvement. This paper describes the tasks, and analyzes techniques proposed by various teams. 1 Introduction In populous countries (e.g., India), pending legal cases have grown exponentially. Due to the na- ture of the legal domain, it may not be possible to automate the entire judicial pipeline completely; nevertheless, many intermediate tasks can be au- tomated to augment legal practitioners and hence expedite the system. For example, legal documents can be processed with the help of Natural Language Processing (NLP) techniques to organize and struc- ture the data to be amenable to automatic search and retrieval. However, legal texts are different from commonly occurring texts typically used to train NLP models. Legal documents are quite long (tens and sometimes hundreds of pages), making it challenging to process (Malik et al., 2021). An- other challenge with legal documents is that they use different lexicons compared to standard texts. Moreover, legal documents are manually typed in countries like India and are highly unstructured and noisy (e.g., spelling and grammatical mistakes). Above mentioned challenges make it difficult to apply existing NLP models and techniques directly, which calls for the development of legal domain- specific techniques. To promote research and development in the area of Legal-AI we proposed shared tasks via “LegalE- val - Understanding Legal Texts". This paper gives details about the shared task and our experience in organizing LegalEval. In total 26 teams submitted systems papers in the shared task. Though the doc- uments pertained to the Indian legal system, teams across the globe participated showing an interest in developing legal-NLP techniques that could later be generalized to other legal systems as well. The paper is organized as follows. In Section 2, we outline …

Content truncated to 500 words