Training ELECTRA Augmented with Multi-word Selection

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 244–255 August 1–6, 2021. ©2021 Association for Computational Linguistics 244 Joint Optimization of Tokenization and Downstream Model Tatsuya Hiraoka†, Sho Takase†, Kei Uchiumi‡, Atsushi Keyaki‡, Naoaki Okazaki† † Tokyo Institute of Technology ‡ Denso IT Laboratory, Inc. {tatsuya.hiraoka, sho.takase}@nlp.c.titech.ac.jp {kuchiumi, akeyaki}@d-itlab.co.jp okazaki@c.titech.ac.jp Abstract Since traditional tokenizers are isolated from a downstream task and model, they cannot out- put an appropriate tokenization depending on the task and model, although recent studies im- ply that the appropriate tokenization improves the performance. In this paper, we propose a novel method to find an appropriate tokeniza- tion to a given downstream model by jointly optimizing a tokenizer and the model. The pro- posed method has no restriction except for us- ing loss values computed by the downstream model to train the tokenizer, and thus, we can apply the proposed method to any NLP task. Moreover, the proposed method can be used to explore the appropriate tokenization for an al- ready trained model as post-processing. There- fore, the proposed method is applicable to var- ious situations. We evaluated whether our method contributes to improving performance on text classification in three languages and machine translation in eight language pairs. Experimental results show that our proposed method improves the performance by deter- mining appropriate tokenizations. 1 Introduction Tokenization, which converts a raw sentence into a sequence of tokens, is a crucial process that affects the performance of NLP tasks. Existing studies have proposed various tokenization methods includ- ing rule-based tokenization (Koehn et al., 2007), dictionary-based tokenization (Kudo, 2006; Morita et al., 2015; Tolmachev et al., 2018; Takaoka et al., 2018), supervised tokenization with neural net- works (Yang et al., 2017; Cai et al., 2017; Yang et al., 2019), and unsupervised tokenization (Gold- water et al., 2006, 2009; Mochihashi et al., 2009; Sennrich et al., 2016; Kudo and Richardson, 2018). Much of prior research has reported that an appro- priate tokenization depends on each downstream task (Xu et al., 2008; Chang et al., 2008; Nguyen et al., 2010; Domingo et al., 2018; Hiraoka et al., 2019; Gowda and May, 2020). Moreover, Hiraoka et al. (2020) implies that we have to consider a downstream model to determine an appropriate to- kenization. In other words, we can improve the performance of a downstream model by determin- ing an appropriate tokenization for the downstream model. However, since traditional tokenizers are isolated from a downstream model, we need to train a given downstream model with each possible tokenization and evaluate its performance to de- termine the appropriate tokenization. Performing such an exploration whenever we construct a new downstream model is impractical. Several studies have addressed the optimization of a tokenizer based on a downstream task or/and model (He et al., 2020; Hiraoka et al., 2020), but existing methods are restricted to specific tasks. He et al. (2020) proposed DPE as a tokenization method for a sequence-to-sequence problem such as machine translation. Their method trains a to- kenizer with a given training corpus, …

Content truncated to 500 words