Revisiting Pre-Trained Models for Chinese Natural Language Processing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 657–668 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 657 Revisiting Pre-trained Models for Chinese Natural Language Processing Yiming Cui1,2, Wanxiang Che1, Ting Liu1, Bing Qin1, Shijin Wang2,3, Guoping Hu2 1Research Center for Social Computing and Information Retrieval (SCIR), Harbin Institute of Technology, Harbin, China 2State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China 3iFLYTEK AI Research (Hebei), Langfang, China 1{ymcui,car,tliu,qinb}@ir.hit.edu.cn 2,3{ymcui,sjwang3,gphu}@iflytek.com Abstract Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre- trained language models. In this paper, we target on revisiting Chinese pre-trained lan- guage models to examine their effectiveness in a non-English language and release the Chi- nese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which im- proves upon RoBERTa in several ways, espe- cially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language mod- els as well as the proposed MacBERT. Ex- perimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with sev- eral findings that may help future research.1 1 Introduction Bidirectional Encoder Representations from Trans- formers (BERT) (Devlin et al., 2019) has become enormously popular and has proven to be effec- tive in recent natural language processing stud- ies, which utilizes large-scale unlabeled training data and generates enriched contextual representa- tions. As we traverse several popular machine read- ing comprehension benchmarks, such as SQuAD (Rajpurkar et al., 2018), CoQA (Reddy et al., 2019), QuAC (Choi et al., 2018), NaturalQuestions (Kwiatkowski et al., 2019), RACE (Lai et al., 2017), we can see that most of the top-performing mod- els are based on BERT and its variants (Dai et al., 2019; Zhang et al., 2019; Ran et al., 2019), demon- strating that the pre-trained language models have 1https://github.com/ymcui/MacBERT become new fundamental components in natural language processing field. Starting from BERT, the community have made great and rapid progress on optimizing the pre- trained language models, such as ERNIE (Sun et al., 2019a), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), SpanBERT (Joshi et al., 2019), AL- BERT (Lan et al., 2019), ELECTRA (Clark et al., 2020), etc. However, training Transformer-based (Vaswani et al., 2017) pre-trained language models are not as easy as we used to train word embed- dings or other traditional neural networks. Typi- cally, training a powerful BERT-large model, which has 24-layer Transformer with 330 million parame- ters, to convergence needs high-memory comput- ing devices, such as TPU, which is very expensive. On the other hand, though various pre-trained lan- guage models have been released, most of them are based on English, and there are few efforts on building powerful pre-trained language models on other languages. In this paper, we aim to build Chinese pre-trained language model …

Content truncated to 500 words