Pooled Contextualized Embeddings for Named Entity Recognition

Proceedings of NAACL-HLT 2019, pages 724–728 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 724 Pooled Contextualized Embeddings for Named Entity Recognition Alan Akbik Zalando Research M¨uhlenstraße 25 10243 Berlin Tanja Bergmann Zalando Research M¨uhlenstraße 25 10243 Berlin {firstname.lastname}@zalando.de Roland Vollgraf Zalando Research M¨uhlenstraße 25 10243 Berlin Abstract Contextual string embeddings are a recent type of contextualized word embedding that were shown to yield state-of-the-art results when utilized in a range of sequence labeling tasks. They are based on character-level language models which treat text as distributions over characters and are capable of generating em- beddings for any string of characters within any textual context. However, such purely character-based approaches struggle to pro- duce meaningful embeddings if a rare string is used in a underspecified context. To ad- dress this drawback, we propose a method in which we dynamically aggregate contextual- ized embeddings of each unique string that we encounter. We then use a pooling oper- ation to distill a global word representation from all contextualized instances. We eval- uate these pooled contextualized embeddings on common named entity recognition (NER) tasks such as CoNLL-03 and WNUT and show that our approach significantly improves the state-of-the-art for NER. We make all code and pre-trained models available to the research community for use and reproduction. 1 Introduction Word embeddings are a crucial component in many NLP approaches (Mikolov et al., 2013; Pen- nington et al., 2014) since they capture latent se- mantics of words and thus allow models to bet- ter train and generalize. Recent work has moved away from the original “one word, one embed- ding” paradigm to investigate contextualized em- bedding models (Peters et al., 2017, 2018; Akbik et al., 2018). Such approaches produce different embeddings for the same word depending on its context and are thus capable of capturing latent contextualized semantics of ambiguous words. Recently, Akbik et al. (2018) proposed a character-level contextualized embeddings ap- Fung B-PER Permadi E-PER ( Taiwan S-LOC ) v Indra S-ORG Figure 1: Example sentence that provides underspecified context. This leads to an underspecified contextual word em- bedding for the string “Indra” that ultimately causes a mis- classification of “Indra” as an organization (ORG) instead of person (PER) in a downstream NER task. proach they refer to as contextual string embed- dings. They leverage pre-trained character-level language models from which they extract hidden states at the beginning and end character positions of each word to produce embeddings for any string of characters in a sentential context. They showed these embeddings to yield state-of-the-art results when utilized in sequence labeling tasks such as named entity recognition (NER) or part-of-speech (PoS) tagging. Underspecified contexts. However, such contex- tualized character-level models suffer from an in- herent weakness when encountering rare words in an underspecified context. Consider the example text segment shown in Figure 1: “Fung Permadi (Taiwan) v Indra”, from the English CONLL-03 test data split (Tjong Kim Sang and De Meulder, 2003). If we consider the word “Indra” to be …

Content truncated to 500 words