SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2405–2412 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2405 SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion Jiahao Chen1,2, Chenjie Cao2, Xiuyan Jiang*1 1 Fudan University, No.825 Zhangheng Road, Pudong New Area, Shanghai, China 2 Pingan Gammalab, No.1119 South Wanpin Road, Xuhui District, Shanghai, China 17212010084@fudan.edu.cn, caochenjie948@pingan.com.cn, *xyjiang@fudan.edu.cn Abstract Pre-trained models have achieved great success in learning unsupervised language representations by self-supervised tasks on large-scale corpora. Recent studies mainly focus on how to fine-tune different downstream tasks from a general pre-trained model. However, some studies show that customized self-supervised tasks for a particular type of downstream task can effectively help the pre-trained model to capture more corresponding knowledge and semantic information. Hence a new pre-training task called Sentence Insertion (SI) is proposed in this paper for Chinese query-passage pairs NLP tasks including answer span prediction, retrieval question answering and sentence level cloze test. The related experiment results indicate that the proposed SI can improve the performance of the Chinese Pre- trained models significantly. Moreover, a word segmentation method called SentencePiece is utilized to further enhance Chinese Bert performance for tasks with long texts. The complete source code is available at https://github.com/ewrfcas/SiBert tensorflow. Keywords: Self-supervised tasks, Bert, Pretrained model 1. Introduction Recently pre-trained language models like Bert (Devlin et al., 2018), XLnet (Yang et al., 2019b), Elmo (Peters et al., 2018) ,GPT (Radford et al., 2018) have been demonstrated to offer substantial performance boosts for many NLP tasks such as Machine Reading Comprehension, Named Entity Recognition, and Natural Language Inference. There are usually two steps in similar models: pre-training and fine- tuning. Model parameters are trained on unlabeled data over different pre-training tasks and then applied to differ- ent labeled downstream tasks for fine-tuning. Researchers were devoted to improving fine-tuning skills to enhance performance of downstream tasks in previous months, but actually some studies (Sun et al., 2019b) (Yang et al., 2019a) (Dong et al., 2019) have shown that self-supervised tasks during pre-training have a huge impact on the perfor- mance of fine-tuning as these tasks determine the learning method and whether pre-trained models can utilize massive unlabeled data efficiently. Bert, the most popular language pre-trained model in NLP communities, includes two pre-training tasks: Masked LM (MLM) and Next Sentence Prediction (NSP). In the MLM task, Bert randomly masks a certain percentage of tokens in the sentences and learns to predict these masked tokens. In the NSP task, Bert learns to predict whether two sentences are adjacent. In fact, the NSP task has two obvious short- comings: (1) Its initial target is to model the relationship between two sentences, which is usually consequential, ad- versative or contradictory. However, due to the long length of two sentences, there are inevitably many domain-specific words. Therefore the Bert model can judge by whether two sentences belong to a same field, not complex inference in sentence level. As a result, the NSP task is more …

Content truncated to 500 words