VART: Vocabulary Adapted BERT Model for Multi-label Document Classification

VART: Vocabulary Adapted BERT Model for Multi-label Document Classification Zhongguang Zheng Fujitsu R&D Center zhengzhg@fujitsu.com Lu Fang Fujitsu R&D Center fanglu@fujitsu.com Jun Sun Fujitsu R&D Center sunjun@fujitsu.com Yiling Cao Fujitsu R&D Center caoyiling@fujitsu.com Abstract Large scale pre-trained language models (PTLMs) such as BERT have been widely used in various natural language process- ing (NLP) tasks, since PTLMs greatly im- prove the downstream task performances by fine-tuning the parameters on the target task datasets. However, in many NLP tasks, such as document classification, the task datasets often contain numerous domain specific words which are not included in the vocabulary of the original PTLM. Those out-of-vocabulary (OOV) words tend to carry useful domain knowledge for the downstream tasks. The do- main gap caused by OOV words may limit the effectiveness of PTLM. In this paper, we present VART, a concise pre-training method to adapt BERT model by learn OOV word representations for multi-label document clas- sification (MLDC) task. VART employs an extended embedding layer to learn the OOV word representations. The extended layer can be pre-trained on the task datasets with high efficiency and low computational resource. The experiments for MLDC task on three datasets from different domains with different sizes demonstrate that VART consistently out- performs the conventional PTLM adaptation methods such as fine-tuning, task adaption and other pre-trained model adaptation methods. 1 Introduction Pre-trained language models (PTLMs) such as GPT (Radford and Narasimhan, 2018) and BERT (Devlin et al., 2018), which are trained on massive unlabeled datasets, can effectively encode rich knowledge into huge parameter spaces. Therefore, by fine-tuning the PTLM parameters, the encoded knowledge is able to benefit a wide range of downstream natural language processing (NLP) tasks (Dai et al., 2021; Liang et al., 2020; Adhikari et al., 2019; Wang et al., 2019; Zhu et al., 2020; Gao et al., 2019). However, applying PTLMs to the specific domain tasks always faces the domain gap problem. Con- ventionally, PTLMs are trained on a large volume of general domain datasets with a fixed vocabulary extracted from the datasets. When applying such general domain PTLMs on a specialized domain dataset, e.g., patent documents, the domain gap be- comes an important factor that hinders the perfor- mance of PTLMs. One of the causes of the do- main gap is the domain words which are not in- cluded in the PLTM vocabulary. Although a PTLM is capable to handle the out-of-vocabulary (OOV) words by splitting each OOV word into multiple in-vocabulary sub-words, for instance, the word “chalcogenide” commonly seen in the patent doc- ument will be split into five sub-words including “ch”, “##al”, “##co”, “##gen” and “##ide” with the vocabulary of BERT-base-cased model. As a result, the representation of “chalcogenide” is di- vided into five embedded vectors. Consequently, the information of “chalcogenide” which is intuitively preferable as an integral representation would be ig- nored in the downstream tasks. Since those OOV words tend to carry rich domain knowledge, the in- formation loss potentially limits the effectiveness of PTLMs for tasks such as multi-label …

Content truncated to 500 words