On Decoding Strategies for Neural Text Generators

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting Google Research, USA {jhclark,dhgarrette,iuliaturc,jwieting}@google.com Abstract Pipelined NLP systems have largely been su- perseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent to- kenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences—without explicit to- kenization or vocabulary—and a pre-training strategy that operates either directly on char- acters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input se- quence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 5.7 F1 on TYDI QA, a challenging multilingual benchmark, despite having fewer model parameters. 1 Introduction End-to-end neural models have generally replaced the traditional NLP pipeline, and with it, the er- ror cascades and feature engineering common to such systems, preferring instead to let the model automatically induce its own sophisticated repre- sentations. Tokenization, however, is one of the few holdovers from that era, with nearly all com- monly used models today requiring an explicit preprocessing stage to segment a raw text string into a sequence of discrete model inputs. Broadly speaking, tokenizers are generally either carefully constructed systems of language-specific rules, CANINE: Character Architecture with No tokenization In Neural Encoders. Code and checkpoints are available on GitHub at http://caninemodel.page.link/code. which are costly, requiring both manual feature engineering and linguistic expertise, or data- driven algorithms such as Byte Pair Encoding (Sennrich et al., 2016), WordPiece (Wu et al., 2016), or SentencePiece (Kudo and Richardson, 2018) that split strings based on frequencies in a corpus, which are less brittle and easier to scale, but are ultimately too simplistic to properly handle the wide range of linguistic phenomena that can’t be captured by mere string-splitting (§2.1). The degree of sophistication required to ac- curately capture the full breadth of linguistic phenomena, along with the infeasibility of writing such rules by hand across all languages and do- mains, suggests that explicit tokenization itself is problematic. In contrast, an end-to-end model that operates directly on raw text strings would avoid these issues, instead learning to compose indi- vidual characters into its own arbitrarily complex features, with potential benefits for both accuracy and ease of use. While this change is conceptu- ally very simple—one could replace the subword vocabulary in a model like BERT (Devlin et al., 2019) with a vocabulary made solely of individ- ual characters—doing so leads to two immediate problems. First, the computational complexity of a transformer (Vaswani et al., 2017), the main com- ponent in BERT as well as other models such as GPT (Radford et al., 2019; Brown et al., 2020) …

Content truncated to 500 words