Does Chinese BERT Encode Word Structure?

Proceedings of the 28th International Conference on Computational Linguistics, pages 2826–2836 Barcelona, Spain (Online), December 8-13, 2020 2826 Does Chinese BERT Encode Word Structure? Yile Wang1,2,3, Leyang Cui1,2,3 and Yue Zhang2,3 1Zhejiang University 2School of Engineering, Westlake University 3Institute of Advanced Technology, Westlake Institute for Advanced Study {wangyile,cuileyang}@westlake.edu.cn yue.zhang@wias.org.cn Abstract Contextualized representations give significantly improved results for a wide range of NLP tasks. Much work has been dedicated to analyzing the features captured by representative models such as BERT. Existing work finds that syntactic, semantic and word sense knowledge are encoded in BERT. However, little work has investigated word features for character-based languages such as Chinese. We investigate Chinese BERT using both attention weight distribution statistics and probing tasks, finding that (1) word information is captured by BERT; (2) word-level features are mostly in the middle representation layers; (3) downstream tasks make different use of word features in BERT, with POS tagging and chunking relying the most on word features, and natural language inference relying the least on such features. 1 Introduction Large scale pre-trained models such as BERT (Devlin et al., 2019) have been widely used in NLP, giv- ing improved results in a wide range of downstream tasks, including dependency parsing (Zhou and Zhao, 2019), summarization (Liu and Lapata, 2019) and reading comprehension (Xu et al., 2019). To better understand the reason behind their effectiveness, a natural question is what knowledge can be learned during the self-supervised pre-training stage. Some previous works prove that BERT effectively captures syntactic information (Goldberg, 2019), semantic information (Jawahar et al., 2019), common- sense knowledge (Cui et al., 2020), and factual knowledge (Petroni et al., 2019) without fine-tuning on task-specific datasets, which explains its effectiveness on related tasks. BERT has also gained success in Chinese NLP (Cui et al., 2019; Sun et al., 2019; Zhang et al., 2019). However, relatively little work investigate knowledge learned by Chinese BERT. Different from English, Chinese sentences are written as sequences of characters without explicit word boundary. Yet Chinese BERT is character-based, where the contextualized representations are built on each character. For traditional Chinese NLP, word segmentation is considered an essential pre-processing step (Cai and Zhao, 2016; Xu and Sun, 2016). In neural modeling, Li et al. (2019) argue that character-based model consistently outperforms word-based model for Chinese, because the former can alleviate the over-fitting and out-of-vocabulary issue. In this paper, we investigate whether Chinese BERT encodes word structure features. We aim to answer the following three research questions. First, how much word information is cap- tured by character-based Chinese BERT? Second, out of 12 representation layers of a BERT encoder, which layers encode the most word-level features? Third, the connection between word-level features embedded in BERT representations and the performance of downstream tasks such as named entity recognition (NER) and natural language inference (NLI). Intuitively, some tasks rely on word features more than other tasks. By analyzing word knowledge inside BERT after fine-tuning on each task, we can gain empirical evidence on how the task is …

Content truncated to 500 words