Evaluating Tokenizers Impact on OOVs Representation with Transformers Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4193–4204 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4193 Evaluating Tokenizers Impact on OOVs Representation with Transformers Models Alexandra Benamar, Cyril Grouin, Meryl Bothua, Anne Vilnat Universit´e Paris-Saclay, CNRS, LISN, Orsay, France {first name}.{last name}@lisn.upsaclay.fr EDF Lab R&D, Palaiseau, France {first name}.{last name}@edf.fr Abstract Transformer models have achieved significant improvements in multiple downstream tasks in recent years. One of the main contributions of Transformers is their ability to create new representations for out-of-vocabulary (OOV) words. In this paper, we have evaluated three categories of OOVs: (A) new domain-specific terms (e.g., “eucaryote” in microbiology), (B) misspelled words containing typos, and (C) cross-domain homographs (e.g., “arm” has different meanings in a clinical trial and anatomy). We use three French domain-specific datasets on the legal, medical, and energetical domains to robustly analyze these categories. Our experiments have led to exciting findings that showed: (1) It is easier to improve the representation of new words (A and B) than it is for words that already exist in the vocabulary of the Transformer models (C), (2) To ameliorate the representation of OOVs, the most effective method relies on adding external morpho-syntactic context rather than improving the semantic understanding of the words directly (fine-tuning) and (3) We cannot foresee the impact of minor misspellings in words because similar misspellings have different impacts on their representation. We believe that tackling the challenges of processing OOVs regarding their specificities will significantly help the domain adaptation aspect of BERT. Keywords: Out-of-vocabulary, Transformer models, Sub-units tokenization, Language models 1. Introduction The most recent NLP models based on neural net- works, such as Long-Short Term Memory (LSTM) net- works (Peters et al., 2018), attentive convolution (Yin and Sch¨utze, 2018) and Transformer models (Vaswani et al., 2017), are trained and evaluated on general- domain datasets. Pre-trained Transformer models such as BERT (Devlin et al., 2018) have proven their ef- fectiveness in adapting to multiple NLP tasks and do- mains. Among the specificities of this architecture, its capacity to adapt to out-of-vocabulary (OOV) terms is a crucial element of its success. The pre-trained models rely on a Unigram tokenizer algorithm (e.g., SentencePiece or WordPiece) (Kudo and Richardson, 2018) or on the Byte-Pair Encoding (BPE) algorithm (Sennrich et al., 2015), which allows the splitting of a word into multiple sub-words. The main idea of dividing words into sub-words is to re- duce the vocabulary size by computing frequent sub- units in OOVs. Unfortunately, these tokenization algo- rithms are purely statistical and lead to semantic infor- mation loss when dealing with domain-specific terms. Bostrom and Durrett (2020) showed that BPE is sub- optimal for language models pre-training, as it did not align well with morphology compared to Unigram for English and Japanese. It suggests that sub-units do not contain semantic or syntactic information and that, consequently, the words are poorly represented in the embedding space. In this study, we study the impact of tokenization on the …

Content truncated to 500 words