Pre-trained Contextualized Character Embeddings Lead to Major Improvements in Time Normalization: a Detailed Analysis

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 68–74 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 68 Pre-trained Contextualized Character Embeddings Lead to Major Improvements in Time Normalization: a Detailed Analysis Dongfang Xu Egoitz Laparra School of InformationSteven Bethard University of Arizona Tucson, AZ {dongfangxu9,laparra,bethard}@email.arizona.edu Abstract Recent studies have shown that pre-trained contextual word embeddings, which assign the same word different vectors in different contexts, improve performance in many tasks. But while contextual embeddings can also be trained at the character level, the effective- ness of such embeddings has not been stud- ied. We derive character-level contextual em- beddings from Flair (Akbik et al., 2018), and apply them to a time normalization task, yield- ing major performance improvements over the previous state-of-the-art: 51% error reduction in news and 33% in clinical notes. We ana- lyze the sources of these improvements, and find that pre-trained contextual character em- beddings are more robust to term variations, infrequent terms, and cross-domain changes. We also quantify the size of context that pre- trained contextual character embeddings take advantage of, and show that such embeddings capture features like part-of-speech and capi- talization. 1 Introduction Pre-trained language models (LMs) such as ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018), OpenAI GPT (Radford et al., 2018), Flair (Akbik et al., 2018) and Bert (Devlin et al., 2018) have shown great improvements in NLP tasks ranging from sentiment analysis to named entity recognition to question answering. These models are trained on huge collections of unlabeled data and produce contextualized word embeddings, i.e., each word receives a different vector representa- tion in each context, rather than a single common vector representation regardless of context as in word2vec (Mikolov et al., 2013) and GloVe (Pen- nington et al., 2014). Research is ongoing to study these models and determine where their benefits are coming from (Peters et al., 2018; Radford et al., 2018; Khandel- wal et al., 2018; Qi et al., 2018; Zhang and Bow- man, 2018). The analyses have focused on word- level models, yet character-level models have been shown to outperform word-level models in some NLP tasks, such as text classification (Zhang et al., 2015), named entity recognition (Kuru et al., 2016), and time normalization (Laparra et al., 2018a). Thus, there is a need to study pre-trained contex- tualized character embeddings, to see if they also yield improvements, and if so, to analyze where those benefits are coming from. All of the pre-trained word-level contextual em- bedding models include some character or sub- word components in their architecture. For ex- ample, Flair is a forward-backward LM trained over characters using recurrent neural networks (RNNs), that generates pre-trained contextual word embeddings by concatenating the forward LM’s hidden state for the word’s last character and the backward LM’s hidden state for the word’s first character. Flair achieves state-of-the-art or compet- itive results on part-of-speech tagging and named entity tagging (Akbik et al., 2018). Though they do not pre-train a LM, Bohnet et …

Content truncated to 500 words