Ginn-Khamov at SemEval-2023 Task 6, Subtask B: Legal Named Entities Extraction for Heterogenous Documents

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 420–425 July 13-14, 2023 ©2023 Association for Computational Linguistics Ginn-Khamov at SemEval-2023 Task 6, Subtask B: Legal Named Entities Extraction for Heterogenous Documents Michael Ginn and Roman Khamov University of Colorado michael.ginn@colorado.edu, roman.khamov@colorado.edu Abstract This paper describes our submission to SemEval-2023 Task 6, Subtask B, a shared task on performing Named Entity Recogni- tion in legal documents for specific legal en- tity types. Documents are divided into the preamble and judgement texts, and certain en- tity types should only be tagged in one of the two text sections. To address this challenge, our team proposes a token classification model that is augmented with information about the docu- ment type, which achieves greater performance than the non-augmented system. 1 Introduction The goal of the Legal NER subtask of the Legal- Eval task is to perform named entity recognition (NER) on English legal texts (Modi et al., 2023). We are interested in detecting entities specific to the legal domain, such as case numbers, as well as differentiating between legal roles for people, such as lawyers and judges. This information can then be used for downstream tasks such as information retrieval. While industry-standard NER systems such as spaCy’s transition-based parser (Honnibal and Johnson, 2015) achieve state-of-the art perfor- mance on most standard NER tasks, it can be de- sirable to have a customizable system for a domain task (Tang et al., 2017). Custom NER systems can support domain-specific entity types and learn to detect entities in structured domain-specific docu- ments. Our primary system utilizes a pretrained lan- guage model with a token classification head. The data provided by Kalamkar et al. (2022) con- sists of annotated sentences from Indian English court judgement documents, which are divided into two sections, the preamble text and judgement text. We observed that while entities of certain classes appear in both the preamble and judgement texts, they should only be identified in one of the two document classes. To address this imbalance, we introduced augmentations that aided our system in making accurate predictions for each document class. We discover that our system is able to achieve similar performance to the baseline on the vali- dation data, and that certain augmentations allow the system to learn more quickly and accurately. Our system shows improvements for certain entity types, although certain entity types remain difficult to label. Our code is available at https://github.com/ michaelpginn/SemEval2023-LegalNER. 2 Background 2.1 Datasets The dataset provided by (Kalamkar et al., 2022) consists of 11.0k training sentences and 1k valida- tion sentences, labeled with fourteen different en- tity types. For example, in the following sentence, "Sri Raja Amareshwar Naik" should be tagged as a RESPONDENT. No one was examined as witness on be- half of other respondents, including re- spondents 2 and Sri Raja Amareshwar Naik. 2.2 Evaluation Metrics The systems are primarily evaluated using the aver- age F1 score over all entity labels, where entities must match exactly to be considered correct. Addi- …

Content truncated to 500 words