Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4238–4248, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4238 Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling Xiaochuang Han and Jacob Eisenstein Georgia Institute of Technology∗ hxc@cmu.edu, me@jacob-eisenstein.com Abstract Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretrain- ing on large corpora of unlabeled text. How- ever, the applicability of this approach is un- known when the target domain varies sub- stantially from the pretraining corpus. We are specifically interested in the scenario in which labeled data is available in only a canon- ical source domain such as newstext, and the target domain is distinct from both the la- beled and pretraining texts. To address this scenario, we propose domain-adaptive fine- tuning, in which the contextualized embed- dings are adapted by masked language mod- eling on text from the target domain. We test this approach on sequence labeling in two challenging domains: Early Modern English and Twitter. Both domains differ substan- tially from existing pretraining corpora, and domain-adaptive fine-tuning yields substantial improvements over strong BERT baselines, with particularly impressive results on out-of- vocabulary words. We conclude that domain- adaptive fine-tuning offers a simple and effec- tive approach for the unsupervised adaptation of sequence labeling to difficult new domains.1 1 Introduction Contextualized word embeddings are becoming a ubiquitous component of natural language pro- cessing (Dai and Le, 2015; Devlin et al., 2019; Howard and Ruder, 2018; Radford et al., 2018; Pe- ters et al., 2018). Pretrained contextualized word embeddings can be used as feature for downstream ∗XH is now at Carnegie Mellon University and JE is now at Google Research. Some of the work was performed while JE was visiting Facebook AI Research. 1Trained models for Early Modern English and Twit- ter are available at https://github.com/xhan77/ AdaptaBERT tasks; alternatively, the contextualized word em- bedding module can be incorporated into an end- to-end system, allowing the embeddings to be fine-tuned from task-specific labeled data. In either case, a primary benefit of contextualized word embeddings is that they seed the learner with distributional information from large unla- beled datasets. However, the texts used to build pretrained con- textualized word embedding models are drawn from a narrow set of domains: • Wikipedia in BERT (Devlin et al., 2019) and ULMFiT (Howard and Ruder, 2018); • Newstext (Chelba et al., 2013) in ELMo (Pe- ters et al., 2018); • BooksCorpus (Zhu et al., 2015) BERT (Devlin et al., 2019) and GPT (Rad-in ford et al., 2018). All three corpora consist exclusively of text writ- ten since the late 20th century; furthermore, Wikipedia and newstext are subject to restrictive stylistic constraints (Bryant et al., 2005).2 It is therefore crucial to determine whether these pre- trained models are transferable to texts from other periods or other stylistic traditions, such as histor- ical documents, …

Content truncated to 500 words