Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6163–6173, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6163 Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models Thuy-Trang Vu♦ Dinh Phung† Faculty of Information Technology, Monash University, Australia ♦{trang.vuthithuy},†{first.last}@monash.edu Gholamreza Haffari† Abstract Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the tar- get task of interest. Current self-supervised adaptation methods are simplistic, as the train- ing signal comes from a small percentage of randomly masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more ef- fectively by allocating self-supervision where it is needed. Furthermore, we propose an ef- fective training strategy by adversarially mask- ing out those tokens which are harder to recon- struct by the underlying MLM. The adversar- ial objective leads to a challenging combina- torial optimisation problem over subsets of to- kens, which we tackle efficiently through re- laxation to a variational lower-bound and dy- namic programming. On six unsupervised do- main adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements. 1 Introduction Contextualised word embedding models are be- coming the foundation of state-of-the-art NLP sys- tems (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Raffel et al., 2019; Brown et al., 2020; Clark et al., 2020). These mod- els are pretrained on large amounts of raw text using self-supervision to reduce the labeled data requirement of target tasks of interest by providing useful feature representations (Wang et al., 2019a). Recent work has shown the importance of further training of pre-trained masked language models (MLMs) on the target domain text, as the benefits of their contextualised representations can deteri- orate substantially in the presence of domain mis- match (Ma et al., 2019; Xu et al., 2019; Wang et al., 2019c; Gururangan et al., 2020). This is partic- ularly crucial in unsupervised domain adaptation (UDA), where there is no labeled data in the target domain (Han and Eisenstein, 2019) and the knowl- edge from source domain labeled data is transferred to the target domain via a common representation space. However, current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of randomly masked-out tokens. Thus, it remains to investigate whether there exist more effective self-supervision strate- gies to bridge the knowledge gap of MLMs about the domains to yield higher-quality adapted mod- els. A key principle of UDA is to learn a common embedding space of both domains which enables transferring a learned model on source task to tar- get task. It is typically done by further pretraining the MLM on a combination of both source and target data. Selecting relevant training examples has been shown to be effective in preventing the negative transfer and boosting the performance of adapted models (Moore and …

Content truncated to 500 words