Pre-training with Meta Learning for Chinese Word Segmentation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5514–5523 June 6–11, 2021. ©2021 Association for Computational Linguistics 5514 Pre-training with Meta Learning for Chinese Word Segmentation Zhen Ke1, Liang Shi1, Songtao Sun1, Erli Meng1, Bin Wang1, Xipeng Qiu2 Xiaomi AI Lab, Xiaomi Inc., Beijing, China1 Shanghai Key Laboratory of Intelligent Information Processing, Fudan University2 {kezhen,shiliang1,sunsongtao,mengerli,wangbin11}@xiaomi.com {xpqiu}@fudan.edu.cn Abstract Recent researches show that pre-trained mod- els (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language model- ing as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model METASEG, which employs a unified architec- ture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empiri- cal results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and sig- nificantly improve model performance in low- resource settings. 1 Introduction Chinese Word Segmentation (CWS) is a fundamen- tal task for Chinese natural language processing (NLP), which aims at identifying word boundaries in a sentence composed of continuous Chinese char- acters. It provides a basic component for other NLP tasks like named entity recognition (Li et al., 2020), dependency parsing (Yan et al., 2020), and seman- tic role labeling (Xia et al., 2019), etc. Generally, most previous studies model the CWS task as a character-based sequence labeling task (Xue, 2003; Zheng et al., 2013; Chen et al., 2015; Ma et al., 2018; Qiu et al., 2020). Recently, pre- trained models (PTMs) such as BERT (Devlin et al., 2019) have been introduced into CWS tasks, which could provide prior semantic knowledge and boost the performance of CWS systems. Yang (2019) di- rectly fine-tunes BERT on several CWS benchmark datasets. Huang et al. (2020) fine-tunes BERT in a Criteria Li Na entered the semi-final CTB6 李娜 进入 半决赛 PKU 李 娜 进入 半 决赛 MSRA 李娜 进入 半 决赛 Table 1: An example of CWS on different criteria. multi-criteria learning framework, where each cri- terion shares a common BERT-based feature extrac- tion layer and has separate projection layer. Meng et al. (2019) combines Chinese character glyph fea- tures with pre-trained BERT representations. Tian et al. (2020) proposes a neural CWS framework WMSEG, which utilizes memory networks to incor- porate wordhood information into the pre-trained model ZEN (Diao et al., 2019). PTMs have been proved quite effective by fine- tuning on downstream CWS tasks. However, PTMs used in previous works usually adopt language modeling as pre-training tasks. Thus, they usu- ally lack task-specific prior knowledge for CWS and ignore the discrepancy between pre-training tasks and downstream CWS tasks. To deal with aforementioned problems of PTMs, we consider introducing a CWS-specific pre- trained model based on existing CWS corpora, to leverage the prior segmentation …

Content truncated to 500 words