Large-Scale Multi-Label Text Classification on EU Legislation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6314 Large-Scale Multi-Label Text Classification on EU Legislation Ilias Chalkidis Manos Fergadiotis Ion AndroutsopoulosProdromos Malakasiotis Department of Informatics, Athens University of Economics and Business, Greece [ihalk,fergadiotis,rulller,ion]@aueb.gr Abstract We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EUR-LEX, annotated with ∼4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Exper- imenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings fur- ther improve performance. We also find that considering only particular zones of the docu- ments is sufficient. This allows us to bypass BERT’s maximum text length limit and fine- tune BERT, obtaining the best results in all but zero-shot learning cases. 1 Introduction Large-scale multi-label text classification (LMTC) is the task of assigning to each document all the relevant labels from a large set, typically contain- ing thousands of labels (classes). Applications include building web directories (Partalas et al., 2015), labeling scientific publications with con- cepts from ontologies (Tsatsaronis et al., 2015), assigning diagnostic and procedure labels to med- ical records (Mullenbach et al., 2018; Rios and Kavuluru, 2018). We focus on legal text process- ing, an emerging NLP field with many applica- tions (e.g., legal judgment (Nallapati and Man- ning, 2008; Aletras et al., 2016), contract element extraction (Chalkidis et al., 2017), obligation ex- traction (Chalkidis et al., 2018)), but limited pub- licly available resources. Our first contribution is a new publicly avail- able legal LMTC dataset, dubbed EURLEX57K, containing 57k English EU legislative documents from the EUR-LEX portal, tagged with ∼4.3k la- bels (concepts) from the European Vocabulary (EUROVOC).1 EUROVOC contains approx. 7k la- bels, but most of them are rarely used, hence they are under-represented (or absent) in EURLEX57K, making the dataset also appropriate for few- and zero-shot learning. EURLEX57K can be viewed as an improved version of the dataset released by Mencia and F¨urnkranzand (2007), which has been widely used in LMTC research, but is less than half the size of EURLEX57K (19.6k documents, 4k EU- ROVOC labels) and more than ten years old. As a second contribution, we experiment with several neural classifiers on EURLEX57K, includ- ing the Label-Wise Attention Network of Mullen- bach et al. (2018), called CNN-LWAN here, which was reported to achieve state of the art perfor- mance in LMTC on medical records. We show that a simpler BIGRU with self-attention (Xu et al., 2015) outperforms CNN-LWAN by a wide mar- gin on EURLEX57K. However, by replacing the CNN encoder of CNN-LWAN with a BIGRU, we ob- tain even better results on EURLEX57K. Domain- specific WORD2VEC (Mikolov et al., 2013) and context-sensitive ELMO embeddings (Peters et al., 2018) yield further improvements. We thus estab- lish strong baselines for EURLEX57K. As a third …

Content truncated to 500 words