Classification of US Supreme Court Cases Using BERT-Based Techniques

Proceedings of Recent Advances in Natural Language Processing, pages 1207–1215 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_128 1207 Classification of US Supreme Court Cases using BERT-Based Techniques Shubham Vatsal New York University, CIMS New York, USA sv2128@nyu.edu Adam Meyers New York University, CIMS New York, USA meyers@cs.nyu.edu John E. Ortega Northeastern University Boston, USA j.ortega@northeastern.edu Abstract Models based on bidirectional encoder repre- sentations from transformers (BERT) produce state of the art (SOTA) results on many nat- ural language processing (NLP) tasks such as named entity recognition (NER), part-of- speech (POS) tagging etc. An interesting phe- nomenon occurs when classifying long docu- ments such as those from the US supreme court where BERT-based models can be considered difficult to use on a first-pass or out-of-the-box basis. In this paper, we experiment with sev- eral BERT-based classification techniques for US supreme court decisions or supreme court database (SCDB) and compare them with the previous SOTA results. We then compare our results specifically with SOTA models for long documents. We compare our results for two classification tasks: (1) a broad classification task with 15 categories and (2) a fine-grained classification task with 279 categories. Our best result produces an accuracy of 80% on the 15 broad categories and 60% on the fine-grained 279 categories which marks an improvement of 8% and 28% respectively from previously reported SOTA results. 1 Introduction Every October, the US supreme court begins a new term on the first Monday. Each term includes a number of significant and complex cases that ad- dress a variety of issues, including environmental protection law, free speech, equal opportunity, tax law etc. Legal court decisions are lawful statements acknowledged by a judge providing the justifica- tion and reasoning for a court ruling. The court’s decisions not only have repercussions on the peo- ple involved in the case but also on the society as a whole. During a regularly scheduled court session, the justice who wrote the main decision sums it up from the bench. A copy of the opinion is then quickly posted on the concerned website. Legal ex- perts must search through and classify these court decisions to support their research. This can re- quire a lot of manual effort. Artificial intelligence (AI) and machine learning (ML) can significantly reduce the burden of this type of manual work. Washington University’s collection of 8419 man- ually labeled supreme court documents (SCDB) (Spaeth et al., 2013) provides the basis for bench marking this type of AI task. There are multiple challenges involved when dealing with SCDB or legal documents of similar kinds. First, these documents are exceptionally long which causes numerous difficulties. For ex- ample, feature vectors can be too large to fit in the memory or contextual variance can be too high for models to handle. Next, the decision of a case re- quires identification of other relevant cases that can support the decision of the current case, which usu- ally involve similar circumstances. Therefore, it is of utmost importance to identify the similarities in …

Content truncated to 500 words