MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

MultiEURLEX – A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer Ilias Chalkidis † ⋄ Manos Fergadiotis ‡ Ion Androutsopoulos ‡ † Department of Computer Science, University of Copenhagen, Denmark ‡ Department of Informatics, Athens University of Economics and Business, Greece ⋄ Cognitiv+ Ltd., London, United Kingdom ilias.chalkidis@di.ku.dk [fergadiotis,ion]@aueb.gr Abstract We introduce MULTI-EURLEX, a new multi- lingual dataset for topic classification of legal documents. The dataset comprises 65k Euro- pean Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the im- portance of chronological, instead of random splits. We use the dataset as a testbed for zero- shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another lan- guage (target). We find that fine-tuning a mul- tilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowl- edge and, consequently, poor zero-shot trans- fer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine- tuning for new end-tasks, help retain multilin- gual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set. 1 Introduction Multilingual learning is an active field of research in NLP. Starting from neural machine translation (Stahlberg, 2020), multilingual neural models are increasingly being considered across NLP tasks and multilingual benchmark datasets for cross-lingual language understanding are becoming available (Hu et al., 2020; Ruder et al., 2021), complement- ing previous monolingual benchmarks (Wang et al., 2018). The initial paradigm of multilingual word embeddings (Ruder et al., 2017) was rapidly ex- panded to pretrained multilingual models (Con- neau et al., 2018), including work on zero-shot cross-lingual transfer (Artetxe and Schwenk, 2019). Multilingual models based on TRANSFORMERs (Vaswani et al., 2017), jointly pretrained on large Figure 1: MULTI-EURLEX covers 23 official EU lan- guages (Table 1) from 7 families (illustrated per EU country in the map). The UK was an EU member un- til 2020. The map should not be taken to imply that no other languages are spoken in EU countries. corpora across multiple languages, have signifi- cantly advanced the state-of-the-art in cross-lingual tasks (Conneau et al., 2020; Xue et al., 2021). In another interesting direction, legal NLP (Ale- tras et al., 2019; Zhong et al., 2020) is an emerging field targeting tasks such as legal judgment pre- diction (Aletras et al., 2016), legal topic classifi- cation (Chalkidis et al., 2019), legal question an- swering (Kim et al., 2015), contract understanding (Hendrycks et al., 2021), to name a few. Generic pretrained language models for legal text in par- ticular have also been introduced (Chalkidis et al., 2020b). But despite rapid growth, cross-lingual transfer has not yet been explored in legal NLP. To facilitate research on cross-lingual transfer for text classification and legal topic classifica- tion in particular, we introduce a new multilingual dataset, MULTI-EURLEX, …

Content truncated to 500 words