Learning Sentence Embeddings In The Legal Domain with Low Resource Settings

Learning Sentence Embeddings in the Legal Domain with Low Resource Settings Sahan Jayasinghe*,1, Lakith Rambukkanage*, Ashan Silva*, Nisansa de Silva*, Amal Shehan Perera*, and Madhavi Perera** *Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka **Parliament of Sri Lanka, Sri Lanka 1sahanjayasinghe.17@cse.mrt.ac.lk Abstract As Natural Language Processing is evolving rapidly, it is used to analyze domain spe- cific large text corpora. Applying Natural Language Processing in a domain with un- common vocabulary and unique semantics requires techniques specifically designed for that domain. The legal domain is such an area with unique vocabulary and semantic interpretations. In this paper we have con- ducted research to develop sentence embed- dings, specifically for the legal domain, to address the domain needs. We have carried this research under two approaches. Due to the availability of a large corpus of raw court case documents, an Auto-Encoder model which re-constructs the input sentence is trained in a self-supervised approach. Pre- trained word embeddings on general corpora and word embeddings specifically trained on legal corpora are also incorporated within the Auto-Encoder. As the next approach we have designed a multitask model with noise discrimination and Semantic Textual Simi- larity tasks. It is expected that these embed- dings and gained insights would help vec- torize legal domain corpora, enabling further application of Machine Learning in the legal domain. 1 Introduction Natural Language Processing (NLP) is advanc- ing rapidly in the research domain as well as in the practical applications. Several researches have been conducted in the recent past, that have made ground breaking progress but some are yet to be discovered and applied in practical applications. It is not trivial to apply ML techniques directly on unstructured data and NLP approaches differ- ent aspects of these problems. Also the advantages of using NLP is best utilized in fields which handle large amounts of textual data. The Legal Domain is such a domain where an abundance of textual data is available, and legal corpora is growing on a daily basis. 1.1 Case Law Case Law documents are one of the aspects which contributes to the rapidly growing textual data in the legal domain. In case law, records of past cases with their evidence arguments and judgment are kept in order to be used as reference and grounds for ongoing cases (cor, 2020). The usage of similar cases with respect to the current case as grounds, is why these documents are very important in a pre- dictive sense. They serve as a good training data source for researches that explore the application of NLP in the Legal Domain. 1.2 Word and Sentence Embeddings NLP consists of many techniques such as parts of speech tagging, sentiment analysis, text gener- ation and language translation among many oth- ers. Regardless, unstructured data requires numer- ical representation to be analysed using ML tech- niques. For this transformation, often times, the text data is converted in to vector format or in other words, embeddings in order to be processed using machine learning and …

Content truncated to 500 words