Сөздерді сегментациялау

The paper reports on a new approach to automatic generation of back-of-book indexes for Chinese books. Parsing on the level of complete sentential analysis is avoided because of the inefficiency …...

This paper describes a set of computer programs for Chinese corpus analysis. These programs include (1) extraction of different characters, bigrams and words; (2) word segmentation based on bigram, maximal-matching …...

The Academia Sinica Balanced Corpus (Sinica Corpus) is the first balanced Chinese corpus with part-of-speech tagging. The corpus (Sinica 2.0) is open to the research community through the WWW (http://www.sinica.edu.twiftms- …...

This paper presents a trainable rule-based algorithm for performing word segmen- tation. The algorithm provides a sim- ple, language-independent alternative to large-scale lexicai-based segmenters requir- ing large amounts of knowledge …...

Palmer ([4]) demonstrated how Brill's Transformation-based Error-Driven Learning can be applied to word segmentation in various languages. We present experimental results which show that such al- gorithms can achieve satisfactory …...

We investigated both English and Chinese ad-hoc information retrieval (IR). Part of our objectives is to study the use of term, phrasal and topical concept level evidence, either individually or …...

One of the most prominent problems in computer processing of the Chinese language is identification of the words in a sentence. Since there are no blanks to mark word boundaries, …...

We present a statistical model of Japanese unknown words consisting of a set of length and spelling models classified by the character types that con- stitute a word. The point …...

The paper presents an entropy-based approach to segment a corpus into words, when no additional information about the corpus or the language, and no other resources such as a lexicon …...

This paper introduces a Chinese summarizier called ThemePicker. Though the system incorporates both statistical and text analysis models, the statistical model plays a major role during the automated process. In …...

In this paper, we propose a method for domain unconstrained language understanding based on the How-net knowledge base. The goal is to construct a system that reads in an article …...

Chinese NE (Named Entity) recognition is a difficult problem because of the uncertainty in word segmentation and flexibility in language structure. This paper proposes the use of a rationality model …...

There is no blank to mark word boundaries in Chinese text. As a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknown words. Conventionally unknown words …...

The paper presents a maximum entropy Chinese character-based parser trained on the Chinese Treebank (“CTB” hence- forth). Word-based parse trees CTB are first converted into character-in based trees, where word-level …...

In various Asian languages, including Chinese, there is no space between words in texts. Thus, most Chinese NLP systems must perform word-segmentation (sentence tokenization). However, successful word-segmentation depends on having …...

Chinese is written without word delimiters so word segmentation is generally considered a key step in processing Chinese texts. This paper presents a new statistical approach to segment Chinese sequences …...

We present a language-independent and unsupervised algorithm for the segmenta- tion of words into morphs. The algorithm is based on a new generative probabilis- tic model, which makes use of …...

Chinese part-of-speech (POS) tagging assigns one POS tag to each word in a Chinese sentence. However, since words are not demarcated in a Chinese sentence, Chinese POS tagging requires word …...

Word segmentation is an important part of many applications, including information retrieval, information filtering, document analysis, and text summarization. In Thai language, the process is complicated since words are written …...

This paper aims to further probe into the problems of ambiguities in automatic identification of determinative-measure compounds (DMs) in Chinese. It is known that Chinese DMs are identifiable by regular …...

We describe a novel approach to statistical machine translation that combines syntactic information in the source language with recent advances in phrasal translation. This method requires a source-language dependency parser, …...

: Automatic term extraction is the first step towards automatic or semi-automatic update of existing domain knowledge base. Most of the researches applied word segmentation as a preproc- essing step …...

. It remains to be a difficult issue to convert Chinese language into Chinese sign language, which makes it hard to implement an obstacle free Chinese sign language information service …...

. Word segmentation for Vietnamese, like for most Asian languages, is an important task which has a significant impact on higher language processing levels. However, it has received little attention …...

Most of the current Chinese word alignment tasks often adopt word segmentation systems firstly to identify words. However, word-mismatching problems exist between languages and will degrade the performance of word …...

Adaptor grammars (Johnson et al., 2007b) are a non-parametric Bayesian extension of Prob- abilistic Context-Free Grammars (PCFGs) which in effect learn the probabilities of en- tire subtrees. In practice, this …...

We introduce a word segmentation ap- proach to languages where word bound- aries are not orthographically marked, with application to Phrase-Based Statis- tical Machine Translation (PB-SMT). In- stead of using …...

The (batch) EM algorithm plays an important role in unsupervised induction, but it some- times suffers from slow convergence. In this paper, we show that online variants (1) provide significant …...

Although a lot of progress has been made recently in word segmentation and POS tagging for Chinese, the output of cur- rent state-of-the-art systems is too inaccu- rate to allow …...

AbstractThis paper first describes an experiment to construct an English-Chinese parallel corpus, then applying the Uplug word alignment tool on the corpus and finally produce and evaluate an English-Chinese word …...

This paper proposes an unsupervised word segmentation algorithm that identi- fies word boundaries in continuous source language text in order to improve the translation quality of statistical machine translation (SMT) …...

We show how features can easily be added to standard generative models for unsuper- vised learning, without requiring complex new training methods. In particular, each component multinomial of a generative …...

In this paper we present a cognitively plau- sible approach to word segmentation that segments in an online fashion using only local information and a lexicon of pre- viously segmented …...

Most existing algorithms for learning latent- variable models—such as EM and existing Gibbs samplers—are token-based, meaning that they update the variables associated with one sentence at a time. The incremental …...

Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that …...

AbstractThe goal of this paper is to investigate French word segmentation strategies using phonemic and lexical transcriptions as well as prosodic and part-of-speech annotations. Average fundamental frequency (f0) profiles and …...

AbstractIn order to utilize the corpus-based techniques that have proven effective in natural language processing in recent years, costly and time-consuming manual creation of linguistic resources is often necessary. Traditionally …...

Bayesian models are usually learned using batch algorithms that have to iterate multiple times over the full dataset. This is both com- putationally expensive and, from a cognitive point of …...

. Numerous studies have analyzed the influences of word segmentation (WS) performance on information retrieval (IR) for Mandarin Chinese and have demonstrated a non-monotonic relationship between WS accuracy and IR …...

During language acquisition, children learn to segment speech into phonemes, syllables, morphemes, and words. We examine word segmentation specifically, and explore the possibility that children might have general- purpose chunking …...

Unlike most Western languages, there are no typographic boundaries between words in written Japanese and Chinese. Word seg- mentation is thus normally adopted as an ini- tial step in most …...

In statistical machine translation, word lattices are used to represent the ambiguities in the preprocessing of the source sentence, such as word segmentation for Chinese or morpholog- ical analysis for …...

Studies of computational models of language acquisition depend to a large part on the input available for experiments. In this paper, we study the effect that input size has on …...

Most previous approaches to syntactic pars- ing of Chinese rely on a preprocessing step of word segmentation, thereby assuming there was a clearly defined boundary between mor- phology and syntax …...

We present a cognitive model of early lexi- cal acquisition which jointly performs word segmentation and learns an explicit model of phonetic variation. We define the model as a Bayesian …...

This paper contributes an approach for expressing non-concatenative morphological phenomena, such as stem derivation Semitic languages, in terms of a mildlyin context-sensitive grammar formalism. This offers a convenient level of …...

Structural information in web text pro- vides natural annotations for NLP prob- lems such as word segmentation and pars- ing. In this paper we propose a discrim- inative learning algorithm …...

For languages where space cannot be a boundary of a word, such as Chinese and Vietnamese, word segmentation is always the task to be done first in a statistical machine …...

AbstractWord Segmentation is usually considered an essential step for many Chinese and Japanese Natural Language Processing tasks, such as name tagging. This paper presents several new observations and analysis on …...

AbstractStress has long been established as a major cue in word segmentation for English infants. We show that enabling a current state-of-the-art Bayesian word segmentation model to take advantage of …...

AbstractIn this paper, we present our attempt at annotating procedural texts with a flow graph as a representation of understanding. The domain we focus on is cooking recipe. The flow …...

AbstractThis paper presents a Japanese-to-English statistical machine translation system specialized for patent translation. Patents are practically useful technical documents, but their translation needs different efforts from general-purpose translation. There are …...

AbstractSynthetic word analysis is a potentially important but relatively unexplored problem in Chinese natural language processing. Two issues with the conventional pipeline methods involving word segmentation are (1) the lack …...

Languages that have no explicit word de- limiters often have to be segmented for sta- tistical machine translation (SMT). This is commonly performed by automated seg- menters trained on manually …...

Stress is a useful cue for English word segmentation. A wide range of computa- tional models have found that stress cues enable a 2-10% improvement in segmen- tation accuracy, depending …...

AbstractThis paper discusses a Thai corpus, TaLAPi, fully annotated with word segmentation (WS), part-of-speech (POS) and named entity (NE) information with the aim to provide a high-quality and sufficiently large …...

AbstractThis paper presents an end-to-end automatic processing system for Arabic. The system performs: correction of common spelling errors pertaining to different forms of alef, ta marbouta and ha, and alef …...

AbstractThis paper aims to examine and evaluate the current development of using Web-as-Corpus (WaC) paradigm in Chinese corpus linguistics. I will argue that the unstable notion of wordhood in Chinese …...

In this paper we propose a framework to improve word segmentation accuracy us- ing input method logs. An input method is software used to type sentences in lan- guages which …...

It is well known that prosodic information is used by infants in early language acquisition. In particular, prosodic boundaries have been shown to help infants with sentence and word- level …...

In this paper, we introduce a new approach for joint segmentation, POS tagging and de- pendency parsing. While joint modeling of these tasks addresses the issue of error prop- agation …...

This paper describes a joint model of word segmentation and phonological alternations, which takes unsegmented utterances as input and infers word segmentations and underlying phonological representations. The model is a …...

AbstractIn recent years there has been a surge of interest in the natural language prosessing related to the real world, such as symbol grounding, language generation, and nonlinguistic data search …...

Word segmentation is a fundamental task for Chinese language processing. Howev- er, with the successive improvements, the standard metric is becoming hard to distin- guish state-of-the-art word segmentation systems. In …...

Word segmentation is usually recognized as the first step for many Chinese natural lan- guage processing tasks, yet its impact on these subsequent tasks is relatively under-studied. For example, how …...

AbstractWe focus on the improvement of accuracy of raw text parsing, from the viewpoint of language resource addition. In Japanese, the raw text parsing is divided into three steps: word …...

AbstractThe fact that Japanese employs scriptio continua, or a writing system without spaces, complicates the first step of an NLP pipeline. Word segmentation is widely used in Japanese language processing, …...

AbstractWe present experiments on word segmentation for Akkadian cuneiform, an ancient writing system and a language used for about 3 millennia in the ancient Near East. To our best knowledge, …...

AbstractIn Sanskrit, the phonemes at the word boundaries undergo changes to form new phonemes through a process called as sandhi. A fused sentence can be segmented into multiple possible segmentations. …...

AbstractParsing Chinese critically depends on correct word segmentation for the parser since incorrect segmentation inevitably causes incorrect parses. We investigate a pipeline approach to segmentation and parsing using word lattices …...

Word segmentation, a fundamental technology for lots of downstream applications, plays a significant role in Natural Language Processing, especially for those languages without explicit delimiters, like Chinese, Korean, Japanese and …...

Most speech and language technologies are trained with massive amounts of speech and text information. However, most of the world languages do not have such resources and some even lack …...

Dependency treebank is an important resource in any language. In this paper, we present our work on building BKTreebank, a dependency treebank for Vietnamese. Important points on designing POS tagset, …...

Depending on the quality of the original document, Optical Character Recognition (OCR) can produce a range of errors – from erroneous letters to additional and spurious blank spaces. We applied …...

This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2.1 corpus (UD2) (Nivre et al., 2017). A software tool, which relies on Elephant …...

AbstractWe describe the first experimental results in neural machine translation for Basque. As a synthetic language featuring agglutinative morphology, an extended case system, complex verbal morphology and relatively free word …...

AbstractWord segmentation is a low-level NLP task that is non-trivial for a considerable number of languages. In this paper, we present a sequence tagging framework and apply it to word …...

AbstractState-of-the-art Natural Language Processing algorithms rely heavily on efficient word segmentation. Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission as well …...

AbstractOne of the basic tasks of computational language documentation (CLD) is to identify word boundaries in an unsegmented phonemic stream. While several unsupervised monolingual word segmentation algorithms exist in the …...

AbstractNeural network models have been actively applied to word segmentation, especially Chinese, because of the ability to minimize the effort in feature engineering. Typical segmentation models are categorized as character-based, …...

AbstractFor languages without natural word boundaries, like Japanese and Chinese, word segmentation is a prerequisite for downstream analysis. For Japanese, segmentation is often done jointly with part of speech tagging, …...

AbstractIn Chinese dependency parsing, the joint model of word segmentation, POS tagging and dependency parsing has become the mainstream framework because it can eliminate error propagation and share knowledge, where …...

AbstractWords are properly segmented in the Persian writing system; in practice, however, these writing rules are often neglected, resulting in single words being written disjointedly and multiple words written without …...

Nowadays, very little research for Indigenous languages has been studied. Indigenous lan- guages bring significant challenges for Natural Language Processing approaches because of multiple features such as polysynthesis, morphological complexity, …...

The prevalence of the web has brought about the construction of many large-scale, automatically segmented and tagged corpora, which inevitably introduces errors due to automation and are likely to have …...

AbstractWord segmentation is a challenging pre-processing step for Thai Natural Language Processing due to the lack of explicit word boundaries. The previous systems rely on powerful neural network architecture alone …...

AbstractWe propose ThaiLMCut, a semi-supervised approach for Thai word segmentation which utilizes a bi-directional character language model (LM) as a way to leverage useful linguistic knowledge from unlabeled data. After …...

While word segmentation is a solved prob- lem in many languages, it is still a chal- lenge in continuous-script or low-resource lan- guages. Like other NLP tasks, word segmen- tation …...

AbstractLexicon information and pre-trained models, such as BERT, have been combined to explore Chinese sequence labeling tasks due to their respective strengths. However, existing methods solely fuse lexicon features via …...

AbstractAspect sentiment triplet extraction (ASTE) is a challenging subtask in aspect-based sentiment analysis. It aims to explore the triplets of aspects, opinions and sentiments with complex correspondence from the context. …...

This paper describes a framework for the creation of new derivational morphology databases for a selected set of productive af- fixes in English. The sample resource ob- tained comprises almost …...

AbstractIn this paper, the Mandarin-English code-switching speech synthesis system has been proposed. To focus on learning the content information between two languages, the training dataset is multilingual artificial dataset whose …...

AbstractThe pre-trained language model has recently dominated most downstream tasks in the NLP area. Particularly, bidirectional Encoder Representations from Transformers (BERT) is the most iconic pre-trained language model among the …...

AbstractThis study aims to evaluate three most popular word segmentation tool for a large Traditional Chinese corpus in terms of their efficiency, resource consumption, and cost. Specifically, we compare the …...

AbstractThis paper introduces PyCantonese, an open-source Python library for Cantonese linguistics and natural language processing. After the library design, implementation, corpus data format, and key datasets included are introduced, the …...

AbstractBoundary information is critical for various Chinese language processing tasks, such as word segmentation, part-of-speech tagging, and named entity recognition. Previous studies usually resorted to the use of a high-quality …...

AbstractWord Segmentation is a fundamental step for understanding Chinese language. Previous neural approaches for unsupervised Chinese Word Segmentation (CWS) only exploits shallow semantic information, which can miss important context. Large …...