LEGAL-BERT: The Muppets straight out of Law School

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2898 LEGAL-BERT: The Muppets straight out of Law School Ilias Chalkidis † ‡ Manos Fergadiotis † ‡ Prodromos Malakasiotis † ‡ Nikolaos Aletras ∗ Ion Androutsopoulos † ‡ † Department of Informatics, Athens University of Economics and Business ‡ Institute of Informatics & Telecommunications, NCSR “Demokritos” ∗ Computer Science Department, University of Sheffield, UK [ihalk,fergadiotis,rulller,ion]@aueb.gr n.aletras@sheffield.ac.uk Abstract BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guide- lines in specialised domains. Here we focus on the legal domain, where we explore sev- eral approaches for applying BERT models to downstream legal tasks, evaluating on multi- ple datasets. Our findings indicate that the previous guidelines for pre-training and fine- tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the avail- able strategies when applying BERT in spe- cialised domains. These are: (a) use the orig- inal BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific cor- pora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we re- lease LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computa- tional law, and legal technology applications. 1 Introduction Pre-trained language models based on Transform- ers (Vaswani et al., 2017), such as BERT (Devlin et al., 2019) and its variants (Liu et al., 2019; Yang et al., 2019; Lan et al., 2019), have achieved state-of-the-art results in several downstream NLP tasks on generic benchmark datasets, such as GLUE (Wang et al., 2018), SQUAD (Rajpurkar et al., 2016), and RACE (Lai et al., 2017). Typically, transfer learning with language mod- els requires a computationally heavy step where the language model is pre-trained on a large corpus and a less expensive step where the model is fine- tuned for downstream tasks. When using BERT, the first step can be omitted as the pre-trained mod- els are publicly available. Being pre-trained on Figure 1: The three alternatives when employing BERT for NLP tasks in specialised domains: (a) use BERT out of the box, (b) further pre-train BERT (FP), and (c) pre-train BERT from scratch (SC). All strategies have a final fine-tuning step. generic corpora (e.g., Wikipedia, Children’s Books, etc.) BERT has been reported to under-perform in specialised domains, such as biomedical or scien- tific text (Lee et al., 2019; Beltagy et al., 2019). To overcome this limitation there are two possible strategies; either further pre-train (FP) BERT on domain specific corpora, or pre-train BERT from scratch (SC) on domain specific corpora. Conse- quently, to employ BERT in specialised domains one may consider three alternative strategies before fine-tuning for the downstream task (Figure 1): (a) use BERT out of the box, (b) further pre-train (FP) BERT on domain-specific corpora, and (c) pre-train BERT from scratch …

Content truncated to 500 words