Neural Word Segmentation with Rich Pretraining

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 839–849 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1078 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 839–849 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1078 Neural Word Segmentation with Rich Pretraining Jie Yang∗and Yue Zhang∗and Fei Dong Singapore University of Technology and Design {jie yang, fei dong}@mymail.sutd.edu.sg yue zhang@sutd.edu.sg Abstract Neural word segmentation research has benefited from large-scale raw texts by leveraging them for pretraining character and word embeddings. On the other hand, statistical segmentation research has ex- ploited richer sources of external informa- tion, such as punctuation, automatic seg- mentation and POS. We investigate the ef- fectiveness of a range of external training sources for neural word segmentation by building a modular segmentation model, pretraining the most important submod- ule using rich external sources. Results show that such pretraining significantly improves the model, leading to accura- cies competitive to the best methods on six benchmarks. 1 Introduction There has been a recent shift of research attention in the word segmentation literature from statisti- cal methods to deep learning (Zheng et al., 2013; Pei et al., 2014; Morita et al., 2015; Chen et al., 2015b; Cai and Zhao, 2016; Zhang et al., 2016b). Neural network models have been exploited due to their strength in non-sparse representation learn- ing and non-linear power in feature combination, which have led to advances in many NLP tasks. So far, neural word segmentors have given compara- ble accuracies to the best statictical models. With respect to non-sparse representation, character embeddings have been exploited as a foundation of neural word segmentors. They serve to reduce sparsity of character ngrams, allowing, for example, “猫(cat) 躺(lie) 在(in) 墙角(corner)” to be connected with “狗(dog) 蹲(sit) 在(in) 墙 ∗ Equal contribution. 角(corner)” (Zheng et al., 2013), which is infeasi- ble by using sparse one-hot character features. In addition to character embeddings, distributed rep- resentations of character bigrams (Mansur et al., 2013; Pei et al., 2014) and words (Morita et al., 2015; Zhang et al., 2016b) have also been shown to improve segmentation accuracies. With respect to non-linear modeling power, var- ious network structures have been exploited to represent contexts for segmentation disambigua- tion, including multi-layer perceptrons on five- character windows (Zheng et al., 2013; Mansur et al., 2013; Pei et al., 2014; Chen et al., 2015a), as well as LSTMs on characters (Chen et al., 2015b; Xu and Sun, 2016) and words (Morita et al., 2015; Cai and Zhao, 2016; Zhang et al., 2016b). For structured learning and inference, CRF has been used for character sequence labelling mod- els (Pei et al., 2014; Chen et al., 2015b) and struc- tural beam search has been used for word-based segmentors (Cai and Zhao, 2016; Zhang et al., 2016b). Previous research has shown that segmentation accuracies can be improved by pretraining charac- ter and word embeddings over large Chinese texts, which is …

Content truncated to 500 words