One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages

Proceedings of Recent Advances in Natural Language Processing, pages 92–101 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_012 92 PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors Andrei-Marius Avram, Vasile P˘ais,, Dan Tufis, Research Institute for Artificial Intelligence, Romanian Academy Bucharest, Romania {andrei.avram,vasile,tufis}@racai.ro Abstract EuroVoc is a multilingual thesaurus that was built for organizing the legislative documen- tary of the European Union institutions. It con- tains thousands of categories at different levels of specificity and its descriptors are targeted by legal texts in almost thirty languages. In this work we propose a unified framework for Eu- roVoc classification on 22 languages by fine- tuning modern Transformer-based pretrained language models. We study extensively the performance of our trained models and show that they significantly improve the results ob- tained by a similar tool - JEX - on the same dataset. The code and the fine-tuned models were open sourced, together with a program- matic interface that eases the process of load- ing the weights of a trained model and of clas- sifying a new document. 1 Introduction EuroVoc1 is a multilingual thesaurus which was originally built up specifically for processing the documentary information of the EU institutions. The covered fields are encompassing both Euro- pean Union and national points of view, with a certain emphasis on parliamentary activities. The current release 4.4 of EuroVoc was published in De- cember 2012 and includes 6,883 IDs for thesaurus concepts (corresponding to the preferred terms), classified into 21 domains (top-level domains), fur- ther refined into 127 subdomains. Additional forms of the preferred terms are also available and are assigned the same ID, subdomains and top-level domains. Multilingual EuroVoc thesaurus descriptors are used by a large number of European Parliaments and Documentation Centres to index their large document collections. The assigned descriptors are 1https://data.europa.eu/data/datasets/ eurovoc then used to search and retrieve documents in the collection and to summarise the document contents for the users. As EuroVoc descriptors exist in one- to-one translations in almost thirty languages, they can be displayed in a language other than the text language and give users cross-lingual access to the information contained in each document. One of the most successful recent approaches in document and text classification involves fine- tuning large pretrained language models on a spe- cific task (Adhikari et al., 2019a; Nikolov and Radi- vchev, 2019). Thus, in this work we propose a tool for classifying legal documents with EuroVoc de- scriptors that uses various flavours of Bidirectional Encoder from Transformers (BERT) (Devlin et al., 2019), specific to each language. We evaluated the performance of our models for each individ- ual language and show that our models obtain a significant improvement over a similar tool - JEX (Steinberger et al., 2012). The The models were further integrated into the RELATE platform (P˘ais, et al., 2020) and an API was provided through the PythonPackage Index (PyPi) interface2 that facil- itates the classification of new documents. The code used to train and evaluate the models was also open-sourced3. The rest of …

Content truncated to 500 words