Development of a Benchmark Corpus to Support Entity Recognition in Job Descriptions

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1201–1208 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1201 Development of a Benchmark Corpus to Support Entity Recognition in Job Descriptions Thomas AF Green, Diana Maynard, Chenghua Lin University of Sheffield / Regent Court 211 Portobello Sheffield S1 4DP, UK {tafgreen1, d.maynard, c.lin}@sheffield.ac.uk Abstract We present the development of a benchmark suite consisting of an annotation schema, training corpus and baseline model for Entity Recognition (ER) in job descriptions, published under a Creative Commons license. This was created to address the distinct lack of resources available to the community for the extraction of salient entities, such as skills, from job descriptions. The dataset contains 18.6k entities comprising five types (Skill, Qualification, Experience, Occupation, and Domain). We include a benchmark CRF-based ER model which achieves an F1 score of 0.59. Through the establishment of a standard definition of entities and training/testing corpus, the suite is designed as a foundation for future work on tasks such as the development of job recommender systems. Keywords: entity recognition, corpus development, job descriptions, natural language processing 1. Introduction The identification and extraction of salient entities is an important task in many real-world information ex- traction applications such as text classification, efficient search algorithms, and content recommendations (Li et al., 2020). In recruitment, job-seekers and recruiting companies alike benefit from systems that automatically and continuously acquire up-to-date information about listed job roles and applicant profiles in terms of skills, qualifications, and experience. In addition, these communities would benefit from investigation into the gap between candidate skills and open positions. This requires tools that can automatically identify and extract skills and related entities from unstructured text data. However, the development of Entity Recognition (ER) models to perform these tasks is severely hindered by the lack of publicly available training data. Many available ER corpora consist of general news articles (Lawson and Eustice, 2010), while information about job descriptions is typically only available on online job portals. Skills, which have no agreed definition in the literature (see Section 2), are often general noun phrases rather than the proper names typically associated with Named Entities, making them harder to detect using gazetteer-based approaches. We define skills explicitly in Section 3.1. To develop better job matching tools, we need to address these problems. We first establish a definition of the relevant entities in order to guide the collection of human-labelled data to be used for training and evaluation of automatic ER tools. Building on existing frameworks (Khobreh et al., 2016), we developed our schema, over several iterations, to include five distinct entities: Skills, Qualifications, Experiences, Domains, and Occupations (see Section 2 for more detail), and use this to build a corpus of annotated UK job descriptions. We then present a benchmark ER model using CRF architecture, which is also freely available and can be used as a baseline. Source code can be found in the associated repository. An ER system trained …

Content truncated to 500 words