MaxMatch-Dropout: Subword Regularization for WordPiece

Proceedings of the 29th International Conference on Computational Linguistics, pages 4864–4872 October 12–17, 2022. 4864 MaxMatch-Dropout: Subword Regularization for WordPiece Tatsuya Hiraoka Tokyo Institute of Technology∗ hiraoka.tatsuya@fujitsu.com Abstract We present a subword regularization method for WordPiece, which uses a maximum match- ing algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a search using the maximum matching algorithm. It realizes finetuning with subword regularization for popular pretrained language models such as BERT-base. The experimental results demonstrate that MaxMatch-Dropout improves the performance of text classification and machine translation tasks as well as other subword regularization methods. Moreover, we provide a comparative analysis of subword reg- ularization methods: subword regularization with SentencePiece (Unigram), BPE-Dropout, and MaxMatch-Dropout. 1 Introduction Subword regularization (Kudo, 2018) is a well- known technique for improving the performance of NLP systems, whereby a model is trained with vari- ous tokenizations that are sampled for each training epoch. This approach provides data augmentation and model robustness against tokenization differ- ences. Kudo (2018) first introduced subword regular- ization using a unigram language model that was included in their tokenization tool, namely Sen- tencePiece (Kudo and Richardson, 2018), and re- ported its effectiveness on machine translation tasks. Provilkov et al. (2020) proposed a subword regularization method for byte pair encoding (BPE) known as BPE-Dropout and demonstrated the su- periority of their method over that using the uni- gram language model in machine translation tasks. Moreover, subword regularization contributes to the performance improvement of text classification tasks (Hiraoka et al., 2019). ∗The author is currently affiliated with Fujitsu Limited. This work was carried out at the Tokyo Institute of Technology. w o r o r r d d d Accepting state Non-accepting state Dropped state Figure 1: MaxMatch-Dropout randomly removes ac- cepting states in the trie. In this figure, a state corre- sponding to “word” is dropped and a single input “word” is tokenized as “w, or, d.” As subword regularization is implemented as a modification of a tokenizer, each method is special- ized to a particular tokenizer type. For example, the original subword regularization (Kudo, 2018) is specialized to a tokenizer that uses the unigram language model and BPE-Dropout is specialized to the BPE-based tokenizer. However, these exist- ing subword regularization tools cannot be directly applied to the other common tokenizers such as WordPiece (Song et al., 2021). WordPiece is a tokenizer that is based on the maximum matching algorithm. It is used as the de- fault tokenizer for the popular pretrained language model BERT (Devlin et al., 2018). Although the widely used BERT models (e.g., BERT-base) can improve the performance of various NLP tasks, subword regularization cannot be used for the fine- tuning of the model because no subword regular- ization method exists for WordPiece. The use of subword regularization for the finetuning of pre- trained models with WordPiece may result in a further performance improvement. In this paper, we present a simple modification of WordPiece for the use of subword regulariza- tion. The proposed method, which is known …

Content truncated to 500 words