Word-level Perturbation Considering Word Length and Compositional Subwords

Findings of the Association for Computational Linguistics: ACL 2022, pages 3268 - 3275 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Word-level Perturbation Considering Word Length and Compositional Subwords Tatsuya Hiraoka†, Sho Takase†, Kei Uchiumi‡, Atsushi Keyaki‡, Naoaki Okazaki† † Tokyo Institute of Technology ‡ Denso IT Laboratory, Inc. {tatsuya.hiraoka, sho.takase}@nlp.c.titech.ac.jp {kuchiumi, akeyaki}@d-itlab.co.jp okazaki@c.titech.ac.jp Abstract We present two simple modifications for word- level perturbation: Word Replacement consid- ering Length (WR-L) and Compositional Word Replacement (CWR). In conventional word re- placement, a word in an input is replaced with a word sampled from the entire vocabulary, re- gardless of the length and context of the target word. WR-L considers the length of a target word by sampling words from the Poisson dis- tribution. CWR considers the compositional candidates by restricting the source of sampling to related words that appear in subword regular- ization. Experimental results showed that the combination of WR-L and CWR improved the performance of text classification and machine translation. 1 Introduction Word-level perturbation is a well-known technique used NLP (Zhang and Yang, 2018; Takase and Kiyono, 2021). For example, word replacement (WR) (Bengio et al., 2015; Zhang and LeCun, 2015) randomly replaces words in the input se- quence with words sampled from a vocabulary. The conventional WR uses a uniform distribution for sampling. Although a simple method, it is as effec- tive as complex methods, such as adversarial per- turbations (Takase and Kiyono, 2021). However, the conventional WR frequently replaces original words with unrelated words. If the probability of replacement (hyperparameter) is set to be large, a perturbed input sequence would be drastically different from the original one, which would sig- nificantly affect performance. Thus, it is important to search for an appropriate hyperparameter. Subword regularization (SR) (Kudo, 2018; Hi- raoka et al., 2019; Provilkov et al., 2020) is an- other effective method for word-level perturbation. We used different tokenizations sampled from a pretrained language model in each training epoch with the SR. As this method focuses only on tok- enization, unrelated words are not used. However, Original Sequence: up / da / tion d updat Substrings Overlapped Subwords 𝑃𝑜(1; 𝜆 = 2) 𝑃𝑜*(1; 𝜆 = 2) 𝑃𝑜(5; 𝜆 = 2) 𝑃𝑜*(2; 𝜆 = 2) 𝑃𝑜*(5; 𝜆 = 2) a at ation 𝑥! 𝑥!" 𝑆"! u p d a < t i o up da tion n > at updat ation Sampling & Replacing Figure 1: Outline of replacing the word “da” in “up/da/tion” using our method, CWR-L. sampling tokenization takes a longer time owing to its complex procedure for managing various tok- enization candidates. In addition, the improvement achieved by SR is sometimes unimpressive in com- parison with WR; however, it requires a consider- able amount of time. In this study, we propose two approaches to com- promise between WR and SR. Our method restricts candidates in WR to related words in terms of (1) word length and (2) tokenization. The first ap- proach weights the distribution for word sampling based on the length of the target …

Content truncated to 500 words