Unsupervised Domain Clusters in Pretrained Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747–7763 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7747 Unsupervised Domain Clusters in Pretrained Language Models Roee Aharoni1 & Yoav Goldberg1,2 1 Computer Science Department, Bar Ilan University 2 Allen Institute for Artificial Intelligence first.last@gmail.com Abstract The notion of “in-domain data” in NLP is of- ten over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addi- tion, domain labels are many times unavail- able, making it challenging to build domain- specific systems. We show that massive pre- trained language models implicitly learn sen- tence representations that cluster by domains without supervision – suggesting a simple data- driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data se- lection methods for neural machine translation across five diverse domains, where they outper- form an established approach as measured by both BLEU and by precision and recall of sen- tence selection with respect to an oracle. 1 Introduction It is common knowledge in modern NLP that us- ing large amounts of high-quality training data is a key aspect in building successful machine-learning based systems. For this reason, a major challenge when building such systems is obtaining data in the domain of interest. But what defines a do- main? Natural language varies greatly across top- ics, styles, levels of formality, genres and many other linguistic nuances (van der Wees et al., 2015; van der Wees, 2017; Niu et al., 2017). This over- whelming diversity of language makes it hard to find the right data for the task, as it is nearly im- possible to well-define the exact requirements from such data with respect to all the aforementioned aspects. On top of that, domain labels are usually unavailable – e.g. in large-scale web-crawled data like Common Crawl1 which was recently used to 1https://commoncrawl.org/ it koran subtitles medical law bert-base-uncased Figure 1: A 2D visualization of average-pooled BERT hidden-state sentence representations using PCA. The colors represent the domain for each sentence. train state-of-the-art pretrained language models for various tasks (Raffel et al., 2019). Domain data selection is the task of selecting the most appropriate data for a domain from a large cor- pus given a smaller set of in-domain data (Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Silva et al., 2018). In this work, we propose to use the recent, highly successful self-supervised pre-trained language models, e.g. Devlin et al. (2019); Liu et al. (2019) for domain data selec- tion. As pretrained LMs demonstrate state-of-the- art performance across many NLP tasks after being trained on massive amounts of data, we hypothe- size that the robust representations they learn can be useful for mapping sentences to domains in an unsupervised, data-driven approach. We show that these models indeed learn …

Content truncated to 500 words