Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3935 - 3940 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Unsupervised Word Segmentation with BERT Oriented Probing and Transformation Wei Li1∗, Yuhan Song2∗, Qi Su3, Yanqiu Shao1 1School of Information Science, Beijing Language and Culture University 2School of EECS, Peking University 3School of Foreign Languages, Peking University liweitj47@blcu.edu.cn {songyuhan, sukia}@pku.edu.cn shaoyanqiu@blcu.edu.cn Abstract Word Segmentation is a fundamental step for understanding many languages. Previous neu- ral approaches for unsupervised Chinese Word Segmentation (CWS) only exploit shallow se- mantic information, which can miss important context. Large scale Pre-trained language mod- els (PLM) have achieved great success in many areas. In this paper, we propose to take advan- tage of the deep semantic information embed- ded in PLM (e.g., BERT) with a self-training manner, which iteratively probes and trans- forms the semantic information in PLM into explicit word segmentation ability. Extensive experiment results show that our proposed ap- proach achieves a state-of-the-art F1 score on two CWS benchmark datasets. The proposed method can also help understand low resource languages and protect language diversity.1 1 Introduction There exist many low resource fields and languages where labeled word segmentation is inaccessible, which makes unsupervised word segmentation de- sirable. Previous unsupervised word segmenta- tion methods mainly apply statistical models to either evaluate the quality of possible segmented sequence with discriminative models (e.g., Mu- tual Information (Chang and Lin, 2003)) or esti- mate the generative probabilities with generative models (e.g., Hidden Markov Model (Chen et al., 2014)). However, these statistical methods can only make use of the limited contextual informa- tion, thus yielding less competitive performance. With the thrive of neural networks, researchers have applied neural models for unsupervised word segmentation. Sun and Deng (2018) propose a segmental language model (SLM) to estimate the ∗Equal Contribution. 1The code is available at https://github. com/liweitj47/BERT_unsupervised_word_ segmentation generative probability with recurrent networks. Al- though SLM can exploit more contextual informa- tion compared with statistical models, it is still weak in modeling deep semantic information, lim- ited by its model capacity and training data scale. Pre-trained language models trained on large scale data have shown superior ability to model contextual information, and achieve great success in various tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019). Inspired by the at- tempt for interpreting BERT (Wu et al., 2020), we propose to take advantage of the semantic repre- sentation ability of BERT to evaluate the closeness between characters in a probing manner. To be more specific, we assume that the difference be- tween masking one character and masking several adjacent characters as a whole reveals the closeness between that character and the adjacent ones. Although this probing-based method can take advantage of the large amount of knowledge em- bedded in BERT, it only implicitly exploits the rep- resentation ability of BERT. To transfer the implicit knowledge into explicit segmentation boundary, we propose to apply a self-training method that trans- forms the segmentation decision from generative methods with …

Content truncated to 500 words