RethinkCWS: Is Chinese Word Segmentation a Solved Task?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5676–5686, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5676 RethinkCWS: Is Chinese Word Segmentation a Solved Task? Jinlan Fu† ∗, Pengfei Liu♯ ∗, Qi Zhang†, Xuanjing Huang† † School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University ♯Carnegie Mellon University {fujl16,qz,xjhuang}@fudan.edu.cn, pliu3@cs.cmu.edu Abstract The performance of the Chinese Word Seg- mentation (CWS) systems has gradually reached a plateau with the rapid development of deep neural networks, especially the suc- cessful use of large pre-trained models. In this paper, we take stock of what we have achieved and rethink what’s left in the CWS task. Methodologically, we propose a fine- grained evaluation for existing CWS systems, which not only allows us to diagnose the strengths and weaknesses of existing models (under the in-dataset setting), but enables us to quantify the discrepancy between differ- ent criterion and alleviate the negative trans- fer problem when doing multi-criteria learning. Strategically, despite not aiming to propose a novel model in this paper, our comprehen- sive experiments on eight models and seven datasets, as well as thorough analysis, could search for some promising direction for future research. We make all codes publicly available and release an interface that can quickly evalu- ate and diagnose user’s models: https://github. com/neulab/InterpretEval. 1 Introduction Chinese word segmentation (CWS), as a crucial first step in Chinese language processing, has drawn a large body of research (Sproat and Shih, 1990; Xue and Shen, 2003; Huang et al., 2007; Liu et al., 2014). Recent years have seen remark- able success in the use of deep neural networks on CWS (Zhou et al., 2017; Yang et al., 2017; Ma et al., 2018; Yang et al., 2019; Zheng et al., 2013; Chen et al., 2015b,a; Cai and Zhao, 2016; Pei et al., 2014), and the large unsupervised pre-trained mod- els drive the state-of-the-art results to a new level (Huang et al., 2019). However, the performance of CWS systems grad- ually reaches a plateau and the development of this ∗These two authors contributed equally. field has slowed down. For example, the CWS sys- tems on many existing datasets (e.g. msr, ctb) have achieved F1-score higher than 97.0 but with little further improvement. Naturally, a question would be raised: is CWS a solved task? When we rethink on what we have achieved so far, we find that there are still some important while rarely dis- cussed unsolved questions for this task: Q1: Does current excellent performance (e.g. more than 98.0 F1-score on the msr dataset) indi- cate a perfect CWS system, or are there still some limitations? Existing CWS systems are mainly eval- uated by a corpus-level metric. The holistic mea- sure fails to provide a fine-grained analysis. As a result, we are not clear about what the strengths and weaknesses of a specific model are. To address this problem, we shift the traditional trend of holistic evaluation to fine-grained evalua- tion, in which the notion of the attribute …

Content truncated to 500 words