ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models Linting Xue∗, Aditya Barua∗, Noah Constant∗, Rami Al-Rfou∗, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel Google Research, USA {lintingx,adityabarua,nconstant,rmyeid,sharannarang,mihirkale,adarob} @google.com,craffel@gmail.com Abstract Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model ar- chitectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architec- ture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, train- ing FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1 1 Introduction An important consideration when designing NLP models is the way that text is represented. One common choice is to assign a unique token ID to each word in a fixed finite vocabulary. A given piece of text is thus converted into a sequence of tokens by a tokenizer before being fed into a model for processing. An issue with using a fixed vocabulary of words is that there is no obvious ∗Equal contribution. 1https://github.com/google-research/byt5. way to process a piece of text that contains an out-of-vocabulary word. A standard approach is to map all unknown words to the same <UNK> token, but this prevents the model from distinguishing between different out-of-vocabulary words. Subword tokenizers (Sennrich et al., 2016; Wu et al., 2016; Kudo and Richardson, 2018) present an elegant solution to the out-of-vocabulary prob- lem. Instead of mapping each word to a single token, subword tokenizers decompose words into smaller subword units with a goal of minimizing the total length of the token sequences for a fixed vocabulary size. As an example, a subword tok- enizer might tokenize the word doghouse as the pair of tokens dog and house even if doghouse is not in the subword vocabulary. This flexibility has caused subword tokenizers to become the de facto way to tokenize text over the past few years. However, subword tokenizers still exhibit var- ious undesirable behaviors. Typos, variants in spelling and capitalization, and morphological changes can all cause the token representation of a word or phrase to change completely, which can result in mispredictions. Furthermore, unknown characters (e.g., from a language that was not used when the subword vocabulary …

Content truncated to 500 words