Investigating the Effect of Lexical Segmentation in Transformer-based Models on Medical Datasets

Investigating the Effect of Lexical Segmentation in Transformer-based Models on Medical Datasets Vincent Nguyen Australian National University CSIRO Data61 vincent.nguyen@anu.edu.au Sarvnaz Karimi CSIRO Data61 Sydney, Australia sarvnaz.karimi@csiro.au Zhenchang Xing Australian National University Canberra, Australia zhenchang.xing@anu.edu.au Abstract Transformer-based models have been popu- lar recently and have improved performance for many Natural Language Processing (NLP) Tasks, including those in the biomedical field. Previous research suggests that, when us- ing these models, an in-domain vocabulary is more suitable than using an open-domain vocabulary. We investigate the effects of a specialised in-domain vocabulary trained from scratch on a biomedical corpus. Our research suggests that, although the in-domain vocabu- lary is useful, it is usually constrained by the corpora size because these models needs to be trained from scratch. Instead, it is more useful to have more data, perform additional pretrain- ing steps with a corpus-specific vocabulary.1 1 Introduction In the natural language processing domain, there is a requirement for a fixed-sized vocabulary during training which could lead to Out-Of-Vocabulary (OOV) problem (Luong et al., 2015). This prob- lem is when the fixed vocabulary model encoun- ters an unseen word during inference, and the model is unable to handle it appropriately. Word- Piece tokenisation, initially used in machine trans- lation systems (Wu et al., 2016), has been widely successful in addressing the OOV problem by seg- menting unseen words into word pieces as a rep- resentation for the unknown word. Previous re- search has either replaced unseen words with a special token (Luong et al., 2015), used charac- ter word embeddings (Labeau and Allauzen, 2017) as a fall-back, or ignored these words completely. These techniques have shortcomings as they do not attempt to represent the unseen word or re- quire additional processing and memory as with character embeddings. WordPiece tokenisation is 1Our code is publicly available at Lexical-Segmentation- Transformer. WordPiece: arthralgias → art-hra-al-gia-s Ideal: arthralgias → arthr-algias arthr- means joints, -algias means pain Figure 1: Word segmentation in WordPiece and the ideal segmentation using medical morphemes. a trade-off, where there is no need for special han- dling of out-of-vocabulary, as unseen words are segmented into sub-word units. It allows a limited vocabulary to represent an infinitely sized vocabu- lary space. Models that successfully use WordPiece tokeni- sation include the transformer-based architectures: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019). BERT uses WordPieces as morphemes to aid the contex- tual representation of words. BERT performs at a state-of-the-art level on the GLUE tasks (Wang et al., 2019) due to its ability to fine-tune specif- ically to each task. Given this success, the model has also been applied to the biomedical domain through models such as BioBERT (Lee et al., 2019), which applies additional pretrain- ing on the MEDLINE and PubMed corpora for biomedical text representation. However, these BioBERT models sometimes do not perform well on biomedical tasks, and in some instances are even worse than vanilla BERT (Zhu et al., 2019; Peng et al., 2019; Nguyen et al., …

Content truncated to 500 words