Chinese Word Segmentation

Automatic identification of Chinese personal names in unrestricted texts is a key task in Chinese word segmentation, and can affect other NLP tasks such as word segmentation and information retrieval, …...

Covering ambiguity is one of the two basic types of ambiguities in Chinese word segmentation. We regard its resolution as equivalent to word sense disambiguation, and make use of the …...

This paper presents an integrated approach for Chinese word segmentation, which can perform disambiguation and unknown word identification simultaneously on the input. In this work, a hybrid model is used …...

In this paper we report results of a supervised machine-learning approach to Chinese word segmentation. A maximum entropy tagger is trained on manually annotated data to automatically assign to Chinese …...

The output of Chinese word segmentation can vary according to different linguistic definitions of words and different engineering requirements, and no single standard can satisfy all linguists and all computer …...

This paper presents a Chinese word segmen- tation system that uses improved source- channel models of Chinese sentence genera- tion. Chinese words are defined as one of the following four …...

This paper presents a Chinese word segmen- tation system which can adapt to different domains and standards. We first present a sta- tistical framework where domain-specific words are identified in …...

Chinese word segmentation is a difficult, im- portant and widely-studied sequence modeling problem. This paper demonstrates the abil- ity of linear-chain conditional random fields (CRFs) to perform robust and accurate …...

During the process of unknown word detection in Chinese word segmentation, many detected word candidates are invalid. These false unknown word candidates deteriorate the overall segmentation accuracy, as it will …...

Markov order-1 conditional random fields (CRFs) and semi-Markov CRFs are two popular models for sequence segmenta- tion and labeling. Both models have ad- vantages in terms of the type of …...

This paper presents a discriminative pruning method of n-gram language model for Chinese word segmentation. To reduce the size of the language model that is used in a Chinese word …...

. This paper proposes an empirical comparison between word-based method and character-based method for Chinese word segmentation. In three Chinese word segmentation Bakeoffs, character-based method quickly rose as a mainstream …...

Standard approaches to Chinese word seg- mentation treat the problem as a tagging task, assigning labels to the characters in the sequence indicating whether the char- acter marks a word …...

Chinese word segmentation (CWS) is a necessary step in Chinese-English statisti- cal machine translation (SMT) and its per- formance has an impact on the results of SMT. However, there are …...

In this paper, we describe a new rerank- ing strategy named word lattice reranking, for the task of joint Chinese word segmen- tation and part-of-speech (POS) tagging. As a derivation …...

Conventional approaches to Chinese word segmentation treat the problem as a character- based tagging task. Recently, semi-Markov models have been applied to the problem, in- corporating features based on complete …...

The character-based tagging approach is a dominant technique for Chinese word segmentation, and both discrimi- native and generative models can be adopted in that framework. However, generative and discriminative charac- …...

AbstractIn Chinese texts, words composed of single or multiple characters are not separated by spaces, unlike most western languages. Therefore Chinese word segmentation is considered an important first step in …...

This paper proposes a novel feature for conditional random field (CRF) model in Chinese word segmentation system. The system uses a conditional random field as machine learning model with one …...

This work represents several unsupervised feature selections based on frequent strings that help improve conditional random fields (CRF) model for Chinese word segmentation (CWS). These features include character-based N-gram (CNG), …...

AbstractWe adopt the corpus-informed approach to example sentence selections for the construction of a reference grammar. In the process, a database containing sentences that are carefully selected by linguistic experts …...

Features for Robust Cross-Domain Chinese Word Segmentation LI Xiaoqing1, WANG Kun1, ZONG Chengqing1 and SU Keh-Yih2 (1) National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, …...

In this paper we first describe the technol- ogy of automatic annotation transformation, which is based on the annotation adaptation algorithm (Jiang et al., 2009). It can auto- matically transform …...

AbstractDomain adaptation is an important topic for natural language processing. There has been extensive research on the topic and various methods have been explored, including training data selection, model combination, …...

We investigate whether suffix related features can significantly improve the performance of character-based approaches for Chinese word segmentation (CWS). Since suffixes are quite productive in forming new words, and OOV …...

Nowadays supervised sequence labeling models can reach competitive performance on the task of Chinese word segmenta- tion. However, the ability of these mod- els is restricted by the availability of …...

Chinese word segmentation and part-of- speech tagging (S&T) are fundamental steps for more advanced Chinese language processing tasks. Recently, it has at- tracted more and more research interests to exploit …...

We address the problem of informal word recognition in Chinese microblogs. A key problem is the lack of word delimiters in Chinese. We exploit this reliance as an opportunity: recognizing …...

Currently most of state-of-the-art methods for Chinese word segmentation (CWS) are based on supervised learning, which depend on large scale annotated corpus. However, these supervised methods do not work well …...

AbstractAnnotated word structures are useful for various Chinese NLP tasks, such as word segmentation, POS tagging and syntactic parsing. Chinese word structures are often represented by binary trees, the nodes …...

AbstractWord Segmentation is usually considered an essential step for many Chinese and Japanese Natural Language Processing tasks, such as name tagging. This paper presents several new observations and analysis on …...

Supervised methods have been the domi- nant approach for Chinese word segmen- tation. The performance can drop signif- icantly when the test domain is different from the training domain. In …...

Automatic extraction of new words is an indispensable precursor to many NLP tasks such as Chinese word segmentation, named entity extraction, and sentimen- t analysis. This paper aims at extract- …...

Structured perceptron becomes popular for various NLP tasks such as tagging and parsing. Practical studies on NLP did not pay much attention to its regularization. In this paper, we study …...

There is rich knowledge encoded in on- line web data. For example, punctua- tion and entity tags in Wikipedia data define some word boundaries in a sen- tence. In this …...

This study investigates on building a better Chinese word segmentation mod- el for statistical machine translation. aims at leveraging word boundary infor-It mation, automatically learned by bilin- gual character-based alignments, …...

We report an empirical investigation on type-supervised domain adaptation for joint Chinese word segmentation and POS-tagging, making use of domain- specific tag dictionaries and only un- labeled target domain data …...

Currently most of state-of-the-art meth- ods for Chinese word segmentation are based on supervised learning, whose fea- tures are mostly extracted from a local con- text. These methods cannot utilize …...

AbstractIn this paper, we propose a new annotation approach to Chinese word segmentation, part-of-speech (POS) tagging and dependency labelling that aims to overcome the two major issues in traditional morphology-based …...

Character-based and word-based methods are two main types of statistical models for Chinese word segmentation, the for- mer exploiting sequence labeling models over characters and the latter typically ex- ploiting …...

AbstractDifferent linguistic perspectives causes many diverse segmentation criteria for Chinese word segmentation (CWS). Most existing methods focus on improve the performance for each single criterion. However, it is interesting to …...

AbstractSocial media provides users a platform to publish messages and socialize with others, and microblogs have gained more users than ever in recent years. With such usage, user profiling is …...

AbstractWe present neural network-based joint models for Chinese word segmentation, POS tagging and dependency parsing. Our models are the first neural approaches for fully joint Chinese analysis that is known …...

AbstractChinese word segmentation (CWS) trained from open source corpus faces dramatic performance drop when dealing with domain text, especially for a domain with lots of special terms and diverse writing …...

This paper presents a novel word granularity-aware annotation framework for Chinese. Anchored in current functionalist linguistics, this model rearranges the boundary of word segmentation and linguistic annotation, and gears toward …...

AbstractNamed entity recognition (NER) in Chinese is essential but difficult because of the lack of natural delimiters. Therefore, Chinese Word Segmentation (CWS) is usually considered as the first step for …...

AbstractSegmenting a chunk of text into words is usually the first step of processing Chinese text, but its necessity has rarely been explored. In this paper, we ask the fundamental …...

AbstractWe investigate subword information for Chinese word segmentation, by integrating sub word embeddings trained using byte-pair encoding into a Lattice LSTM (LaLSTM) network over a character sequence. Experiments on standard …...

AbstractChinese word segmentation and dependency parsing are two fundamental tasks for Chinese natural language processing. The dependency parsing is defined at the word-level. Therefore word segmentation is the precondition of …...

AbstractWord-level information is important in natural language processing (NLP), especially for the Chinese language due to its high linguistic complexity. Chinese word segmentation (CWS) is an essential task for Chinese …...

AbstractTaking greedy decoding algorithm as it should be, this work focuses on further strengthening the model itself for Chinese word segmentation (CWS), which results in an even more fast and …...

AbstractChinese word segmentation (CWS) and part-of-speech (POS) tagging are two fundamental tasks for Chinese language processing. Previous studies have demonstrated that jointly performing them can be an effective one-step solution …...

AbstractChinese word segmentation (CWS) and part-of-speech (POS) tagging are important fundamental tasks for Chinese language processing, where joint learning of them is an effective one-step solution for both tasks. Previous …...

AbstractThe performance of the Chinese Word Segmentation (CWS) systems has gradually reached a plateau with the rapid development of deep neural networks, especially the successful use of large pre-trained models. …...

AbstractBreaking domain names such as openresearch into component words open and research is important for applications like Text-to-Speech synthesis and web search. We link this problem to the classic problem …...

AbstractThe ambiguous annotation criteria lead to divergence of Chinese Word Segmentation (CWS) datasets in various granularities. Multi-criteria Chinese word segmentation aims to capture various annotation criteria among datasets and leverage …...

Pre-trained language models (e.g., BERT) sig- nificantly alleviate two traditional challeng- ing problems for Chinese word segmentation (CWS): segmentation ambiguity and out-of- vocabulary (OOV) words. However, such im- provements are …...

Chinese word segmentation (CWS) is a fun- damental task for Chinese information pro- cessing, which always suffers from out-of- vocabulary word issues, especially when it is tested on data from …...

Chinese word segmentation and part-of- speech tagging are necessary tasks in terms of computational linguistics and application of natural language processing. Many re- searchers still debate the demand for Chinese …...

AbstractRecent state-of-the-art (SOTA) effective neural network methods and fine-tuning methods based on pre-trained models (PTM) have been used in Chinese word segmentation (CWS), and they achieve great results. However, previous …...

Chinese word segmentation has entered the deep learning era which greatly reduces the hassle of feature engineering. Recently, some researchers attempted to treat it as character- level translation, which further …...

AbstractThis paper proposes to cast end-to-end span-based SRL as a word-based graph parsing task. The major challenge is how to represent spans at the word level. Borrowing ideas from research …...

AbstractThe evolution of language follows the rule of gradual change. Grammar, vocabulary, and lexical semantic shifts take place over time, resulting in a diachronic linguistic gap. As such, a considerable …...

AbstractWord Segmentation is a fundamental step for understanding Chinese language. Previous neural approaches for unsupervised Chinese Word Segmentation (CWS) only exploits shallow semantic information, which can miss important context. Large …...

AbstractChinese Word Segmentation (CWS) intends to divide a raw sentence into words through sequence labeling. Thinking in reverse, CWS can also be viewed as a process of grouping a sequence …...