AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4692–4700 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4692 AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain Jimin Hong∗, Taehee Kim∗, Hyesu Lim∗and Jaegul Choo Korea Advanced Institute of Science and Technology (KAIST) {jimmyh, taeheekim, hyesulim, jchoo}@kaist.ac.kr Abstract During the fine-tuning phase of transfer learn- ing, the pretrained vocabulary remains un- changed, while model parameters are updated. The vocabulary generated based on the pre- trained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabu- lary by expanding it with domain-specific vo- cabulary based on a tokenization statistic. Fur- thermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pre- trained language model with a regularization term. Our method achieved consistent perfor- mance improvements on diverse domains (i.e., biomedical, computer science, news, and re- views). 1 Introduction A language model (LM) is pretrained with a large corpus in a general domain and then is fine-tuned to perform various downstream tasks, such as text classification, named entity recognition, and ques- tion answering. However, fine-tuning the LM is challenging when the downstream domain is signif- icantly different from the pretrained domain, requir- ing domain adaptation to improve the downstream performance [Gururangan et al., 2020, Lee et al., 2020, Beltagy et al., 2019]. Prior approaches conducted additional training with a large domain-specific corpus in between pretraining and fine-tuning. In these approaches, the pretrained vocabulary remains unchanged, al- though the model is being adapted to a downstream domain, such as biomedicine or politics. We argue that the vocabulary should also be adapted during the fine-tuning process towards downstream data. Recent studies (e.g., SciB- ERT [Beltagy et al., 2019]) showed that using an ∗Equal Contribution Model Parameters Vocabulary Fine-tuning Downstream data (a) Fine-tuning only model parameters (b) Fine-tuning both model parameters & vocabulary (AVocaDo) (c) Examples of tokenization Domain Word Tokenized with (A) Tokenized with (B) bluetooth blue ##tooth bluetooth corticosterone co ##rti ##cos ##ter ##one cor ##tic ##osterone disrespectful di ##sr ##es ##pe ##ct ##ful disrespect ##ful Pretrained Model Parameters Vocabulary Fine-tuning Downstream data Pretrained Model Parameters Adapted Vocabulary Fine-tuned Model Parameters Fine-tuned (A) (B) Fig1 Current Version Figure 1: Overview of AVocaDo. AVocaDo updates the vocabulary (b) not only fine-tuning the model parame- ters as done by previous approaches (a). fine-tuning the vocabulary has benefit on tokenizing domain-specific words (c). optimized vocabulary for a particular downstream domain is more effective than using the vocabulary generated in pretraining stage. However, these ap- proaches required a large domain-specific corpus additional to the downstream data in order to con- struct optimized vocabulary for the downstream domain. We propose to Adapt the Vocabulary to down- stream Domain (AVocaDo), which updates the pre- trained vocabulary by expanding it with words from the downstream data without requiring additional domain-specific corpus. The relative importance of words is considered in determining …

Content truncated to 500 words