A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products

Proceedings of the 29th International Conference on Computational Linguistics, pages 1014–1023 October 12–17, 2022. 1014 A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products Kesong Liu∗and Jianhui Jiang and Feifei Lyu Alibaba Group {kesong.lks,huidong.jjh,lvfeifei.lff}@alibaba-inc.com Abstract We present a biomedical knowledge enhanced pre-trained language model for medicinal prod- uct vertical search. Following ELECTRA’s replaced token detection (RTD) pre-training, we leverage biomedical entity masking (EM) strategy to learn better contextual word repre- sentations. Furthermore, we propose a novel pre-training task, product attribute prediction (PAP), to inject product knowledge into the pre-trained language model efficiently by lever- aging medicinal product databases directly. By sharing the parameters of PAP’s transformer encoder with that of RTD’s main transformer, these two pre-training tasks are jointly learned. Experiments demonstrate the effectiveness of PAP task for pre-trained language model on medicinal product vertical search scenario, which includes query-title relevance, query in- tent classification, and named entity recogni- tion in query. 1 Introduction Pre-trained language models (PLMs) have signif- icantly improved the performance of various nat- ural language processing (NLP) tasks in the re- cent years. It is now a common practice to adapt the pretrain-then-finetune approach in NLP. PLMs such as BERT (Devlin et al., 2019) and ELEC- TRA (Clark et al., 2020) capturing word meaning through self-supervised learning from large corpus have shown a significant improvement on various text mining tasks. In the biomedical domain, many BERT variants such as BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2021) and BioMedBERT (Chakraborty et al., 2020), follow either continual pre-training or pre- training from scratch approach using domain spe- cific corpora to further improve the model perfor- mance. Our motivation is to apply powerful pre-trained language models on medicinal product vertical ∗Corresponding author search engine to solve query understanding and query title relevance tasks. Most of the models trained on either the general corpus or the medical literature corpus lack medicinal product knowledge. For the product vertical search scenario, medici- nal product information is usually stored in struc- tured relational database tables, and traditional pre- trained language models focus on natural language text in the form of sentences without considering the semantic relationship modeling of structured text in the product information tables. From the perspective of users’ search habits, in addition to searching directly for the medicine name, users also often search for disease, symptoms and other im- portant proudct attribute words to find the medicine. Therefore, we propose a novel pre-training task called product attribute prediction (PAP). Although we could carefully craft a medicinal product knowl- edge graph from product databases and then try to inject the extracted explicit knowledge graph into the pre-trained language model, we argue that the procedure of building product knowledge graph is laborious but avoidable by training the language model directly on product structural information. In this paper, we propose a novel ELEC- TRA(Clark et al., 2020) based biomedical knowl- edge enhanced pre-trained language model. It con- sists of two …

Content truncated to 500 words