UO-LouTAL at SemEval-2023 Task 6: Lightweight Systems for Legal Processing

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1412–1420 July 13-14, 2023 ©2023 Association for Computational Linguistics UO-LouTAL at SemEval-2023 Task 6: Lightweight Systems for Legal Processing Bosch S´ebastien University of Orl´eans (France) Est`eve Louis University of Orl´eans (France) Loo Joanne University of Orl´eans (France) Minard Anne-Lyse LLL-CNRS, University of Orl´eans (France) anne-lyse.minard@univ-orleans.fr Abstract This paper presents the work produced by stu- dents of the University of Orl´eans Masters in Natural Language Processing program by way of participating in SemEval Task 6, LegalE- val, which aims to enhance the capabilities of legal professionals through automated sys- tems. Two out of the three sub-tasks avail- able – Rhetorical Role prediction (RR) and Legal Named Entity Recognition (L-NER) – were tackled, with the express intent of devel- oping lightweight and interpretable systems. For the L-NER sub-task, a CRF model was trained, augmented with post-processing rules for some named entity types. A macro Fβ=1 score of 0.74 was obtained on the DEV set, and 0.64 on the evaluation set. As for the RR sub-task, two sentence classification systems were built: one based on the Bag-of-Words technique with L-NER system output inte- grated, the other using a sentence-transformer approach. Rule-based post-processing then converted the results of the sentence classifica- tion systems into RR predictions. The better- performing Bag-of-Words system obtained a macro Fβ=1 score of 0.49 on the DEV set and 0.57 on the evaluation set. 1 Introduction The ever-increasing volume of legal documents in populous countries such as India, for example, shows that automated systems may prove to be highly beneficial. Automating such processes re- duces the need to perform unrewarding tasks such as manual information retrieval, thereby facilitat- ing the work of legal professionals, while reserving more complex aspects of law and decisions for their intervention ; and shortens the overall time required for legal processing (Kalamkar et al., 2022b). Two out of the three LegalEval 2023 (Modi et al., 2023) sub-tasks available were tackled: the Rhetor- ical Roles (RR) textual classification task (sub-task A) and the Legal-Named Entities Extraction (L- NER) task (sub-task B). Systems for the sub-tasks were not developed in silo: some RR systems made use of data that was first pre-processed with L-NER system-generated annotations, which in some cases improved perfor- mance on the DEV set by approximately 2 points 1. Some challenges encountered include (but are not limited to) handling complex and precise se- mantics developed over long spans of text in the RR sub-task; and understanding the sub- tleties distinguishing the five named entity types pertaining to persons (JUDGE, OTHER PERSON, PETITIONER, RESPONDENT, WITNESS) for the L-NER sub-task. Special attention was placed on developing lightweight and interpretable systems. The ex- press focus on being lightweight was mainly to curb the carbon footprint of machine learning sys- tems and improve widespread usability as state-of- the-art architectures such as transformers (Vaswani et al., 2017) consume considerable computational resources and energy (Strubell et al., 2019), release undesirable pollutants such as greenhouse gases, and is less …

Content truncated to 500 words