ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11871–11890 July 9-14, 2023 ©2023 Association for Computational Linguistics ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain Mike Zhang☼ and Rob van der Goot☼ and Barbara Plank☼� ☼Department of Computer Science, IT University of Copenhagen, Denmark �MaiNLP, Center for Information and Language Processing, LMU Munich, Germany {mikz, robv}@itu.dk b.plank@lmu.de Abstract The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-Rlarge, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence la- beling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-Rlarge on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level. 1 Introduction The dynamic nature of labor markets, driven by technological changes, migration, and digitization, has resulted in a significant amount of job advertise- ment data (JAD) being made available on various platforms to attract qualified candidates (Brynjolf- sson and McAfee, 2011, 2014; Balog et al., 2012). This has led to an increase in tasks related to JAD, including skill extraction (Kivimäki et al., 2013; Zhao et al., 2015; Sayfullina et al., 2018; Smith et al., 2019; Tamburri et al., 2020; Shi et al., 2020; Chernova, 2020; Bhola et al., 2020; Zhang et al., 2022a,b,c; Green et al., 2022; Gnehm et al., 2022; Beauchemin et al., 2022; Decorte et al., 2022; Goyal et al., 2023), skill classification (Decorte et al., 2022; Zhang et al., 2022b), job title classification (Javed et al., 2015, 2016; Decorte et al., 2021; Green et al., 2022), de-identification of entities in job postings (Jensen et al., 2021), and multilingual skill entity linking (ESCO, 2022). While some previous studies have focused on JAD in non-English languages (Zhang et al., 2022b; Gnehm et al., 2022; Beauchemin et al., 2022), their baselines have typically relied on language-specific models, either using domain-adaptive pre-training (DAPT; Gururangan et al., 2020) or off-the-shelf models. The lack of comprehensive, open-source JAD data in various languages makes it difficult to fully pre-train a language model (LM) using such data. In this work, we seek external resources that can help improve the multilingual performance on the JAD domain. We use the ESCO taxon- omy (le Vrang …

Content truncated to 500 words