From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3605–3620 July 9-14, 2023 ©2023 Association for Computational Linguistics From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding Li Sun1†, Florian Luisier2, Kayhan Batmanghelich1, Dinei Florencio2, Cha Zhang2 1Boston University 2Microsoft 1{lisun,batman}@bu.edu, 2{flluisie,dinei,chazhang}@microsoft.com Abstract Current state-of-the-art models for natural lan- guage understanding require a preprocessing step to convert raw text into discrete tokens. This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes. This fixed vocabulary limits the model’s robustness to spelling errors and its capacity to adapt to new domains. In this work, we introduce a novel open-vocabulary language model that adopts a hierarchical two-level ap- proach: one at the word level and another at the sequence level. Concretely, we design an intra- word module that uses a shallow Transformer architecture to learn word representations from their characters, and a deep inter-word Trans- former module that contextualizes each word representation by attending to the entire word sequence. Our model thus directly operates on character sequences with explicit awareness of word boundaries, but without biased sub-word or word-level vocabulary. Experiments on var- ious downstream tasks show that our method outperforms strong baselines. We also demon- strate that our hierarchical model is robust to textual corruption and domain shift. 1 Introduction Pre-trained language models with Transformers have achieved breakthroughs in many natural lan- guage processing (NLP) tasks (Devlin et al., 2019; Liu et al., 2019). One of the key advantages of Transformers over traditional feature-engineered NLP pipelines is that Transformers enable end- to-end training from vast amount of data to au- tomatically learn the optimal language representa- tion (Mikolov et al., 2013b). However, most re- cent language models still require a separate pre- processing stage known as tokenization. Tokeniza- tion is a process that splits raw text parts into a list of discrete tokens from a fixed vocabulary. This †Work done during internship at Microsoft. PyTorch-like pseudocode is available in Appendix A.3. Word BERT Tokens Spelling Errors changable (changeable) ch, ##anga, ##ble (change, ##able) outragous (outrageous) out, ##rag, ##ous (outrage, ##ous) Domain Shift reimbursement re, ##im, ##bur, ##se, ##ment invoice in, ##vo, ##ice Table 1: Examples of sub-word tokenization of mis- spelled words (with their correct spellings in parenthe- ses) and words from domain-specific corpus (e.g. busi- ness documents). Pre-trained sub-word tokenizer tends to over-segment words from these two categories, which produces less meaningful tokens. pre-defined vocabulary remains as an important bottleneck preventing truly end-to-end training of language models (Tay et al., 2021; Islam et al., 2022). Based on the granularity of the basic token units, tokenization methods can be divided into three cat- egories: character-based, subword-based and word- based. A word-based tokenizer segments sentence into smaller chunks of words. Due to language complexity and memory limit, a word-based vocab- ulary can not represent all possible words. Word- level tokenization thus frequently runs into the is- sue of out-of-vocabulary words. A character-based tokenizer simply splits the …

Content truncated to 500 words