Cross-Domain Language Modeling: An Empirical Investigation

Cross-Domain Language Modeling: An Empirical Investigation Vincent Nguyen1,2 Sarvnaz Karimi1 Maciej Rybinski1 Zhenchang Xing2 1CSIRO Data61, Sydney, Australia 2The Australian National University, Canberra, Australia {firstname.lastname}@csiro.au {zhenchang.xing}@anu.edu.au Abstract Transformer encoder models exhibit strong performance in single-domain applications. However, in a cross-domain situation, using a sub-word vocabulary model results in sub- word overlap. This is an issue when there is an overlap between sub-words that share no semantic similarity between domains. We hy- pothesize that alleviating this overlap allows for a more effective modeling of multi-domain tasks; we consider the biomedical and general domains in this paper. We present a study on reducing sub-word overlap by scaling the vo- cabulary size in a Transformer encoder model while pretraining with multiple domains. We observe a significant increase in downstream performance in the general-biomedical cross- domain from a reduction in sub-word overlap. 1 Introduction Contemporary language models are pretrained on massive, linguistically diverse corpora (Lan et al., 2020; Devlin et al., 2019a). It is not uncommon for these models to excel at benchmark downstream tasks (Wang et al., 2019a), given the use of con- textual representations (Devlin et al., 2019b) that are trained on a variety of source domains—a term used to describe a distribution of language on a given topic or genre (for example BIOMEDICAL, SCIENTIFIC)—or GENERAL domain. However, the benefit of GENERAL domain pretraining for spe- cialized application is questionable, as applying these language models (Gu et al., 2020) to special- ized tasks is worse than using specialized coun- terparts (Beltagy et al., 2019). This degradation still occurs after sequential pretraining on special- ized domains (Shin et al., 2020) when fine-tuned (updates to pretraining) to downstream tasks. We hypothesize some of this degradation lies in the use of a sub-word vocabulary (Si et al., 2019). Sub-word vocabularies (Sennrich et al., 2016; Wu et al., 2016a,b) allow for efficient modeling of a source language distribution with a limited vocab- ulary size. However, problematically sub-words can be shared between different words—for exam- ple hypotension and hypocritical—with different meanings. This potentially conflates the vector rep- resentation of a sub-word (or wordpiece) causing sub-word overlap. When this overlap occurs with sub-words appearing in multiple domain contexts we call this cross-domain sub-word overlap. As a pilot empirical study, we investigate reduc- ing cross-domain sub-word overlap, by increasing vocabulary size, in language models pretrained in the GENERAL and BIOMEDICAL cross-domain. To evaluate the effect of sub-word overlap, general and biomedical domain benchmarks are used in this study as the task distribution includes differ- ent linguistic phenomena such as grammar, sen- timent, textual similarity, natural language infer- ence (Wang et al., 2019a). Interestingly, we find that disjoint sub-word vocabulary sets are not ideal. Some sub-word overlap is necessary and unavoid- able, and a different level of overlap is ideal for each target domain. We also find a positive trend occurs when reducing cross-domain sub-word over- lap, suggesting that there is a trade-off depending on the target downstream task and domain. To better understand the results, we look at the …

Content truncated to 500 words