Tokenisation
method
ID: tokenisation
Related Papers
This Phrase-Based SMT System is Out of Order: Generalised Word Reordering in Machine Translation
Many natural language processes have some degree of preprocessing of data: tokenisation, stemming and so on. In the domain of Statisti- cal Machine Translation it has been shown that word …...
Character Mapping and Ad-hoc Adaptation: Edinburgh’s IWSLT 2020 Open Domain Translation System
AbstractThis paper describes the University of Edinburgh’s neural machine translation systems submitted to the IWSLT 2020 open domain Japanese↔Chinese translation task. On top of commonplace techniques like tokenisation and corpus …...
Exploring the Language of Data
AbstractWe set out to uncover the unique grammatical properties of an important yet so far under-researched type of natural language text: that of short labels typically found within structured datasets. …...