Токенизация
task
ID: tokenizing
Related Papers
Use of Mutual Information Based Character Clusters in Dictionary-less Morphological Analysis of Japanese
For languages whose character set is very large and whose orthography does not require spac- ing between words, such as Japanese, tokenizing and part-of-speech tagging are often the diffi- cult …...
On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis
AbstractText preprocessing is often the first step in the pipeline of a Natural Language Processing (NLP) system, with potential impact in its final performance. Despite its importance, text preprocessing has …...