TeamShakespeare at SemEval-2023 Task 6: Understand Legal Documents with Contextualized Large Language Models

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 517–525 July 13-14, 2023 ©2023 Association for Computational Linguistics TeamShakespeare at SemEval-2023 Task 6: Understand Legal Documents with Contextualized Large Language Models Xin Jin∗ Northwest Polytechnical Unviersity xin.jinsw@gmail.com Yuchen Wang∗ Ohio State University wang.9298@osu.edu Abstract The growth of pending legal cases in populous countries, such as India, has become a major is- sue. Developing effective techniques to process and understand legal documents is extremely useful in resolving this problem. In this pa- per, we present our systems for SemEval-2023 Task 6: understanding legal texts (Modi et al., 2023). Specifically, we first develop the Legal- BERT-HSLN model that considers the com- prehensive context information in both intra- and inter-sentence levels to predict rhetorical roles (subtask A) and then train a Legal-LUKE model, which is legal-contextualized and entity- aware, to recognize legal entities (subtask B). Our evaluations demonstrate that our designed models are more accurate than baselines, e.g., with an up to 15.0% better F1 score in subtask B. We achieved notable performance in the task leaderboard, e.g., 0.834 micro F1 score, and ranked No.5 out of 27 teams in subtask A. 1 Introduction The growing amount of legal cases and documents requires more and more human efforts to process them. (Kalamkar et al., 2022b; Malik et al., 2021) In some countries, such as India, legal cases have accumulated in an incredible number. For example, India has more than 47 million cases pending in the courts (Kalamkar et al., 2022b). This has cre- ated a need for automated methods to help judges efficiently understand and process relevant and re- liable information. In addition, these methods can also help students, legal scholars, and court offi- cials who deal with legal documents daily. One way to assist them is to automatically understand and highlight the key information and context of long legal documents. However, understanding legal documents by ma- chines is not an easy task. First, legal documents are often full of technical terminologies, which *These authors contributed equally to this work. can span different legal divisions (Kalamkar et al., 2022b). Furthermore, legal documents can be specific to special cases, such as health (Young, 2009), IT technology (Lu et al., 2017b), and cy- ber security (Shackelford et al., 2016; Jin et al., 2018; Lu et al., 2017a), which will involve domain- specific words. In addition, legal documents can be extremely long (Kalamkar et al., 2022b), which makes dependency-based techniques and models, such as RNN models, fail to extract the con- text information due to gradient vanishes. Fi- nally, typos and unstructured documents introduce noises (Kalamkar et al., 2022b), which makes auto- mated natural language processing challenging. Despite these challenges, predicting rhetorical roles and recognizing named entities in legal docu- ments are very useful for automating the processing and understanding of legal documents. Rhetorical role prediction segments texts and structures noisy legal documents into topically and semantically co- herent units (Ghosh and Wyner, 2019). Named en- tity recognition helps identify key legal entities …

Content truncated to 500 words