Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2068–2077 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2068 Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision Mieradilijiang Maimaiti1,3,4, Yang Liu∗ 1,2,3,4,5, Yuanhang Zheng1,3,4, Gang Chen1,3,4, Kaiyu Huang6, Ji Zhang7, Huanbo Luan1,3,4, Maosong Sun1,3,4,5, 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Beijing National Research Center for Information Science and Technology 5Beijing Academy of Artificial Intelligence 6 School of Computer Science, Dalian University of Technology 7Alibaba DAMO Academy {meadljmm15,zheng-yh19,cg18}@mails.tsinghua.edu.cn {liuyang2011,sms}@tsinghua.edu.cn; luanhuanbo@gmail.com kaiyuhuang@mail.dlut.edu.cn; zj122146@alibaba-inc.com Abstract Recent state-of-the-art (SOTA) effective neu- ral network methods and fine-tuning methods based on pre-trained models (PTM) have been used in Chinese word segmentation (CWS), and they achieve great results. However, pre- vious works focus on training the models with the fixed corpus at every iteration. The inter- mediate generated information is also valuable. Besides, the robustness of the previous neural methods is limited by the large-scale annotated data. There are a few noises in the annotated corpus. Limited efforts have been made by pre- vious studies to deal with such problems. In this work, we propose a self-supervised CWS approach with a straightforward and effective architecture. First, we train a word segmenta- tion model and use it to generate the segmen- tation results. Then, we use a revised masked language model (MLM) to evaluate the quality of the segmentation results based on the pre- dictions of the MLM. Finally, we leverage the evaluations to aid the training of the segmenter by improved minimum risk training. Experi- mental results show that our approach outper- forms previous methods on 9 different CWS datasets with single criterion training and mul- tiple criteria training and achieves better ro- bustness1. 1 Introduction In extensive natural language processing (NLP) sce- narios, most of the tasks are based on word-level methods. When we deal with the Chinese language, ∗Corresponding author 1Code and dataset can be found at https://github. com/miradel51/Self_Supervised_CWS Figure 1: Word segmentation with self-supervision. Our work leverages the reward generated by the pre- dictor to assist the training of the segmenter. there is no specific boundary between two Chinese words. The situation is different in western lan- guages. For instance, there is a space between two words. Thus, Chinese word segmentation (CWS) is considered an essential task, which will accurately represent semantic information of Chinese NLP tasks. Besides, the length of the sentence is short- ened by word segmentation. The shorter length of a sentence is effective for the deep learning method in some cases. Recently, good performance for CWS has al- ready been achieved in large-scale annotated cor- pora as reported by related research (Huang and Zhao, 2007; Zhao et al., 2019). Most methods start with data-driven to improve the performance for CWS. For instance, some neural methods try to incorporate external resources to achieve good per- formance for in-domain and cross-domain CWS (Zhou et al., …

Content truncated to 500 words