Tokenisation

method ID: tokenisation

Related Papers

This Phrase-Based SMT System is Out of Order: Generalised Word Reordering in Machine Translation

Authors: nan

Publication: TIPSTER (2006)

DOI: nan

Many natural language processes have some degree of preprocessing of data: tokenisation, stemming and so on. In the domain of Statisti- cal Machine Translation it has been shown that word …...

28719

Character Mapping and Ad-hoc Adaptation: Edinburgh’s IWSLT 2020 Open Domain Translation System

Authors: nan

Publication: TIPSTER (2020)

DOI: nan

AbstractThis paper describes the University of Edinburgh’s neural machine translation systems submitted to the IWSLT 2020 open domain Japanese↔Chinese translation task. On top of commonplace techniques like tokenisation and corpus …...

20545

Exploring the Language of Data

Authors: nan

Publication: TIPSTER (2020)

DOI: nan

AbstractWe set out to uncover the unique grammatical properties of an important yet so far under-researched type of natural language text: that of short labels typically found within structured datasets. …...

3787