Optimizing Word Segmentation for Downstream Task

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1341–1351 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1341 Optimizing Word Segmentation for Downstream Task Tatsuya Hiraoka†, Sho Takase†, Kei Uchiumi‡, Atsushi Keyaki‡, Naoaki Okazaki† † Tokyo Institute of Technology ‡ Denso IT Laboratory, Inc. {tatsuya.hiraoka, sho.takase}@nlp.c.titech.ac.jp {kuchiumi, akeyaki}@d-itlab.co.jp okazaki@c.titech.ac.jp Abstract In traditional NLP, we tokenize a given sen- tence as a preprocessing, and thus the tok- enization is unrelated to a target downstream task. To address this issue, we propose a novel method to explore a tokenization which is ap- propriate for the downstream task. Our pro- posed method, optimizing tokenization (Op- Tok), is trained to assign a high probability to such appropriate tokenization based on the downstream task loss. OpTok can be used for any downstream task which uses a vector rep- resentation of a sentence such as text classifi- cation. Experimental results demonstrate that OpTok improves the performance of sentiment analysis and textual entailment. In addition, we introduce OpTok into BERT, the state-of- the-art contextualized embeddings and report a positive effect. 1 Introduction Tokenization is a fundamental problem in natural language processing (NLP). We must split a given sequence into a sequence of words for languages that do not contain obvious boundaries, such as Chinese and Japanese. In addition, it is also better to explore appropriate segmentations for languages containing obvious boundaries indicated by whites- paces, such as English (Peng and Dredze, 2015, 2016; Sennrich et al., 2016; He and Sun, 2017; A and Augenstein, 2020; Bollegala et al., 2020). In traditional NLP, we tokenize a given sentence as a preprocessing. Thus, as shown in Figure 1(a), we apply an existing tokenizer to the given sen- tence, and then input the tokenized sentence into a model for a target downstream task. In the con- ventional approach, we obtain the most plausible tokenized sentence based on the tokenizer; how- ever, some studies have varied the tokenization using a sampling during the training to enable the downstream model to adapt to various tokeniza- tions (Kudo, 2018; Hiraoka et al., 2019; Provilkov text text … text text … Tokenizer Corpus Tokenized corpus ~~ Irreversible Preprocessing Downstream Model (a) Conventional Irreversible Tokenization text text … Tokenizer Corpus Optimization to improve the downstream task Downstream Model (b) Optimizing Tokenization (proposed) Figure 1: Overview of (a) conventional tokenization and (b) optimizing tokenization proposed herein. We directly optimize the tokenizer to improve the perfor- mance of the model for a downstream task using the loss of the target task. et al., 2019). Although such a strategy makes the downstream model robust, little attention has been paid to optimizing the tokenizers for a downstream task. Thus, if we acquire an appropriate tokeniza- tion to a downstream task, we might improve the task performance. By contrast, some studies have used multiple to- kenized sentences to prevent the damage depending on the tokenization (Chen et al., 2017; Zhang and Yang, 2018; Yang et al., 2018). Their methods com- pute various tokenizations for a given …

Content truncated to 500 words