Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models

Proceedings of Recent Advances in Natural Language Processing, pages 568–577 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_062 568 Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models Shun Kiyono Sho Takase Shengzhe Li Toshinori Sato LINE Corporation shun.kiyono@linecorp.com, sho.takase@linecorp.com, shengzhe.li@linecorp.com, toshinori.sato@linecorp.com Abstract Pretrained language models require the use of consistent segmentation (e.g., subword- or character-level segmentation) in pretraining and finetuning. In NLP, many tasks are mod- eled by subword-level segmentation better than by character-level segmentation. However, be- cause of their format, several tasks require the use of character-level segmentation. Thus, in order to tackle both types of NLP tasks, lan- guage models must be independently pretrained for both subword and character-level segmen- tation. However, this is an inefficient and costly procedure. Instead, this paper proposes a method for training a language model with uni- fied segmentation. This means that the trained model can be finetuned on both subword- and character-level segmentation. The principle of the method is to apply the subword reg- ularization technique to generate a mixture of subword- and character-level segmentation. Through experiment on BERT models, we demonstrate that our method can halve the com- putational cost of pretraining. 1 Introduction The use of large pretrained language models (PLMs) has become the dominant approach for tackling NLP tasks and applications (Devlin et al., 2019; Bommasani et al., 2021; Kaneko et al., 2020; Konno et al., 2021). One notable characteristic of these models is that the segmentation algorithm must be determined before pretraining the model. Given a pretrained model, users are expected to employ a consistent segmentation algorithm. For example, a common convention is to use a family of subword-level segmentation algo- rithms (Sennrich et al., 2016; Kudo, 2018; Song et al., 2021) with a sufficiently large vocabulary; for example, 8k (Kiyono et al., 2019), 30k (De- vlin et al., 2019), 50k (Radford et al., 2019), or 現状の計算機資源では、全然足りない。 Gold (Current computing resources are far from sufficient.) 現状 の 計算 機 資源 では全然 足りない Subword Segmentation Task Input 現状の計算機資源では全然足りない No segmentation: cannot insert comma 👎 Character Segmentation 現 状 の 計 算 機 資 源 で は 全 然 足 り な い Comma can be inserted between “は” and “全” 👍 Figure 1: Overview of punctuation restoration. Character-level segmentation must be used to insert a missing comma in a given input sentence. Pretraining Corpus Character- segmentation Character PLM Subword- segmentation Subword PLM Previous Work Our Work Character&Subword PLM BPE-dropout Figure 2: Overview of our method. Previously, subword- and character-level pretraining were conducted indepen- dently (left). Conversely, in our method, BPE-dropout enables the training of the language model with unified segmentation (right). 250k (Scao et al., 2022). The subword-level seg- mentation is usually preferred over the character- level segmentation, because subword models of- ten outperform character models (Libovick´y et al., 2022) and are more computationally efficient (Xue et al., 2022). However, such predetermined subword-level seg- mentation may cause a segmentation incompatibil- ity problem, depending on the target downstream task. More specifically, this problem occurs when the pretrained model uses …

Content truncated to 500 words