Is Word Segmentation Necessary for Deep Learning of Chinese Representations?

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3242–3252 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3242 Is Word Segmentation Necessary for Deep Learning of Chinese Representations? Yuxian Meng∗♠, Xiaoya Li∗♠, Xiaofei Sun♠, Qinghong Han♠ Arianna Yuan♠,♥, and Jiwei Li♠,♣ ♣ School of Information, Renmin University of China ♥ Computer Science Department, Stanford University ♠ Shannon.AI { yuxian meng, xiaoya li, xiaofei sun, qinghong han , jiwei li}@shannonai.com xfyuan@stanford.edu Abstract Segmenting a chunk of text into words is usu- ally the first step of processing Chinese text, but its necessity has rarely been explored. In this paper, we ask the fundamental question of whether Chinese word segmentation (CWS) is necessary for deep learning-based Chinese Natural Language Processing. We bench- mark neural word-based models which rely on word segmentation against neural char-based models which do not involve word segmenta- tion in four end-to-end NLP benchmark tasks: language modeling, machine translation, sen- tence matching/paraphrase and text classifica- tion. Through direct comparisons between these two types of models, we find that char- based models consistently outperform word- based models. Based on these observations, we conduct com- prehensive experiments to study why word- based models underperform char-based mod- els in these deep learning-based NLP tasks. We show that it is because word-based models are more vulnerable to data sparsity and the presence of out-of-vocabulary (OOV) words, and thus more prone to overfitting. We hope this paper could encourage researchers in the community to rethink the necessity of word segmentation in deep learning-based Chinese Natural Language Processing. 1 1 Introduction There is a key difference between English (or more broadly, languages that use some form of the Latin alphabet) and Chinese (or other languages that do not have obvious word delimiters such as Korean and Japanese) : words in English can be easily recognized since the space token is a good approxi- mation of a word divider, whereas no word divider 1Yuxian Meng and Xiaoya Li contributed equally to this paper. is present between words in written Chinese sen- tences. This gives rise to the task of Chinese Word Segmentation (CWS) (Zhang et al., 2003; Peng et al., 2004; Huang and Zhao, 2007; Zhao et al., 2006; Zheng et al., 2013; Zhou et al., 2017; Yang et al., 2017, 2018). In the context of deep learning, the segmented words are usually treated as the ba- sic units for operations (we call these models the word-based models for the rest of this paper). Each segmented word is associated with a fixed-length vector representation, which will be processed by deep learning models in the same way as how En- glish words are processed. Word-based models come with a few fundamental disadvantages, as will be discussed below. Firstly, word data sparsity inevitably leads to overfitting and the ubiquity of OOV words limits the model’s learning capacity. Particularly, Zipf’s law applies to most languages including Chinese. Frequencies of many Chinese words are extremely small, making the model impossible …

Content truncated to 500 words