Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining

Findings of the Association for Computational Linguistics: EACL 2023, pages 2516–2526 May 2-6, 2023 ©2023 Association for Computational Linguistics Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining Inderjeet Nair and Natwar Modani Adobe Research, India {inair,nmodani}@adobe.com Abstract Pretraining large language models has resulted in tremendous performance improvement for many natural language processing (NLP) tasks. While for non-domain specific tasks, such mod- els can be used directly, a common strategy to achieve better performance for specific do- mains involves pretraining these language mod- els over domain specific data using objectives like Masked Language Modelling (MLM), Au- toregressive Language Modelling, etc. While such pretraining addresses the change in vocab- ulary and style of language for the domain, it is otherwise a domain agnostic approach. In this work, we investigate the effect of incorporating pretraining objectives that explicitly tries to ex- ploit the domain specific language characteris- tics in addition to such MLM based pretraining. Particularly, we examine two distinct charac- teristics associated with the legal domain and propose pretraining objectives modelling these characteristics. The proposed objectives target improvement of token-level feature represen- tation, as well as aim to incorporate sentence level semantics. We demonstrate superiority in the performance of the models pretrained using our objectives against those trained using domain-agnostic objectives over several legal downstream tasks. 1 Introduction Pre-trained language models exhibit superior per- formance in several NLP tasks. Most of the promi- nent language models optimized over Masked lan- guage modelling with BERT-like (Devlin et al., 2019; Liu et al., 2019b; He et al., 2020) archi- tecture using large unlabelled corpus to achieve state of the art results across many NLP tasks. While the sentence-level tasks like paraphrase de- tection (El Desouki and Gomaa, 2019) and senti- ment analysis (Zhang et al., 2018) benchmarks the capability of the model in effectively modeling the holistic representation of the sentence(s), the token- level tasks like named entity recognition (Li et al., 2020) attempted to assess the quality of contex- tualized token embeddings furnished by the mod- els. However, direct application of these models to domain-specific downstream tasks yields sub- optimal performance (Lee et al., 2020), perhaps due to change in vocabulary and style of language. To overcome this limitation, most commonly used approach involves pre-training a language model over domain specific corpora. For instance, PubMedBERT (Gu et al., 2021) and LEGAL- BERT (Chalkidis et al., 2020) achieved state-of- the-art results for the biomedical and legal domain specific tasks respectively by pre-training over do- main specific corpus using a domain agnostic objec- tive. In this paper, we argue that the performance of these models can be further improved by em- ploying pre-training objectives that exploit the lan- guage characteristics of the domain. We examine two distinct language characteristics of the legal domain, propose pre-training objectives and finally demonstrate superior performance over domain- specific NLP tasks. Legal domain departs from the generic corpora in terms of specialized vocab- ulary, particularly formal syntax, domain-specific knowledge semantics, etc. to the extent that it can be classified …

Content truncated to 500 words