An Empirical Study of Automatic Chinese Word Segmentation for Spoken Language Understanding and Named Entity Recognition

Proceedings of NAACL-HLT 2016, pages 238–248, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics An Empirical Study of Automatic Chinese Word Segmentation for Spoken Language Understanding and Named Entity Recognition Wencan Luo∗ University of Pittsburgh Pittsburgh, PA 15260 wencan@cs.pitt.edu Fan Yang Nuance Communications, Inc. Seattle, WA 98104 fan.yang@nuance.com Abstract Word segmentation is usually recognized as the first step for many Chinese natural lan- guage processing tasks, yet its impact on these subsequent tasks is relatively under-studied. For example, how to solve the mismatch prob- lem when applying an existing word seg- menter to new data? Does a better word seg- menter yield a better subsequent NLP task per- formance? In this work, we conduct an ini- tial attempt to answer these questions on two related subsequent tasks: semantic slot filling in spoken language understanding and named entity recognition. We propose three tech- niques to solve the mismatch problem: us- ing word segmentation outputs as additional features, adaptation with partial-learning and taking advantage of n-best word segmentation list. Experimental results demonstrate the ef- fectiveness of these techniques for both tasks and we achieve an error reduction of about 11% for spoken language understanding and 24% for named entity recognition over the baseline systems. 1 Introduction Unlike English text in which sentences are se- quences of words separated by white spaces, in Chi- nese text (as are some other languages including Arabic, Japanese, etc.), sentences are represented as strings of characters without similar natural delim- iters. Therefore, it is generally claimed that the first step in a Chinese language processing task is to iden- tify the sequence of words in a sentence and mark ∗Work done at Nuance during an internship. boundaries in appropriate places, which is refereed to as the task of Chinese Word Segmentation (CWS). (1) Input: 能穿多少穿多少 CWS 1: 能|穿|多 多 多少 少 少|穿|多少1 (Put on as much clothes as possible.) CWS 2: 能|穿|多|少 少 少|穿|多|少 (Put on as little clothes as possible.) Word segmentations in Chinese text do reduce ambiguities. In the example (1), the same span of text (the input) can convey entirely opposite mean- ings (the English sentences in parentheses) depend- ing on how word boundaries (CWS 1 and CWS 2) are labeled. Therefore, it is generally believed that more accurate word segmentations should benefit more the subsequent Chinese language processing tasks, such as part-of-speech tagging, named entity recognition, etc. There has been quite a number of research in the field of CWS to improve segmen- tation accuracy, yet its impact on the subsequent processing is relatively under-studied. Chang et al. (2008) explore how word segmentation improves machine translation; and Ni and Leung (2014) ex- plore how word segmentation impacts automatic speech recognition yet do not have conclusive find- ings. In this research, we aim to better understand how CWS benefits the subsequent NLP tasks, using semantic slot filling in spoken language understand- ing (SLU) and named entity recognition (NER) as two case studies. In particular, we investigate the impact of Chinese word …

Content truncated to 500 words