Unigram
method
ID: unigram
Related Papers
Fertility Models for Statistical Natural Language Understanding
Several recent efforts in statistical nat- ural language understanding (NLU) have focused on generating clumps of English words from semantic meaning concepts (Miller et al., 1995; Levin and Pierac- cini, …...
Probabilistic and Rule-Based Tagger of an Inflective Language- a Comparison
We present results of probabilistic tag- ging of Czech texts in order to show how these techniques work for one of the highly morphologically ambiguous inflective lan- guages. After description …...
Improving Tokenisation by Alternative Treatment of Spaces
Tokenisation is the first step in almost all NLP tasks, and state-of-the-art transformer-based language models all use subword tokenisation algorithms to process input text. Existing al- gorithms have problems, often …...