Evaluation of Transfer Learning and Domain Adaptation for Analyzing German-Speaking Job Advertisements

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3892–3901 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3892 Evaluation of Transfer Learning and Domain Adaptation for Analyzing German-Speaking Job Advertisements Ann-Sophie Gnehm, Eva B¨uhlmann, Simon Clematide Department of Sociology, Department of Computational Linguistics University of Zurich gnehm@soziologie.uzh.ch, buehlmann@soziologie.uzh.ch, simon.clematide@cl.uzh.ch Abstract This paper presents text mining approaches on German-speaking job advertisements to enable social science research on the development of the labour market over the last 30 years. In order to build text mining applications providing information about profession and main task of a job, as well as experience and ICT skills needed, we experiment with transfer learning and domain adaptation. Our main contribution consists in building language models which are adapted to the domain of job advertisements, and their assessment on a broad range of machine learning problems. Our findings show the large value of domain adaptation in several respects. First, it boosts the performance of fine-tuned task-specific models consistently over all evaluation experiments. Second, it helps to mitigate rapid data shift over time in our special domain, and enhances the ability to learn from small updates with new, labeled task data. Third, domain-adaptation of language models is efficient: With continued in-domain pre-training we are able to outperform general-domain language models pre-trained on ten times more data. We share our domain-adapted language models and data with the research community. Keywords: Text Mining, Transfer Learning, Domain adaptation, BERT, Computational Social Science, Job Advertise- ments 1. Introduction Large-scale automated text analysis is becoming more and more standard in digital scholarship. For com- mon text mining tasks, for instance, Named Entity Recognition and Linking on newspaper texts or Sen- timent Analysis of social media content, several off- the-shelf solutions exist either as research software or commercial services. However, for many specific re- search questions, application domains and languages, specialized natural language processing (NLP) models still need to be trained. Effective training of high-quality text mining models without much annotation effort is now possible thanks to transfer learning, which is typically divided in a pre- training and fine-tuning phase. Over the last years, this machine learning technique, which makes clever use of large amounts of raw text for pre-training, evolved successfully: starting with static lexical word embed- dings as word2vec (Mikolov et al., 2013) followed first by contextualized embeddings encoded by recur- rent neural network-based conditional generative lan- guage models as ELMo (Peters et al., 2018), and then shortly after introducing BERT embeddings (Devlin et al., 2019) where representation learning relies on a transformer-based denoising language modelling pre- training task. Although minor architectural changes have been applied to the original BERT idea, this ap- proach is still state of the art in NLP. The expected benefits of transfer learning approaches are twofold: First, given the pre-trained text represen- tations, fine-tuning needs much less task-specific train- ing material to achieve high performance. Second, fine- tuned models are more robust and less brittle, thus often avoiding …

Content truncated to 500 words