Lexicon Infused Phrase Embeddings for Named Entity Resolution

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 78–86, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Lexicon Infused Phrase Embeddings for Named Entity Resolution Alexandre Passos, Vineet Kumar, Andrew McCallum School of Computer Science University of Massachusetts, Amherst {apassos,vineet,mccallum}@cs.umass.edu Abstract Most state-of-the-art approaches for named-entity recognition (NER) use semi supervised information in the form of word clusters and lexicons. Recently neural network-based language models have been explored, as they as a byprod- uct generate highly informative vector representations for words, known as word embeddings. In this paper we present two contributions: a new form of learn- ing word embeddings that can leverage information from relevant lexicons to improve the representations, and the first tosystem to use neural word embeddings achieve state-of-the-art results on named-entity recognition in both CoNLL and Ontonotes NER. Our system achieves an F1 score of 90.90 on the test set for CoNLL 2003—significantly better than any previous system trained on public data, and matching a system employing massive private industrial query-log data. 1 Introduction In many natural language processing tasks, such as named-entity recognition or coreference reso- lution, syntax alone is not enough to build a high performance system; some external source of in- formation is required. In most state-of-the-art systems for named-entity recognition (NER) this knowledge comes in two forms: domain-specific lexicons (lists of word types related to the de- sired named entity types) and word representa- tions (either clusterings or vectorial representa- tions of word types which capture some of their syntactic and semantic behavior and allow gener- alizing to unseen word types). Current state-of-the-art named entity recogni- tion systems use Brown clusters as the form of word representation (Ratinov and Roth, 2009; Turian et al., 2010; Miller et al., 2004; Brown et al., 1992), or other cluster-based representations computed from private data (Lin and Wu, 2009). While very attractive due to their simplicity, gen- erality, and hierarchical structure, Brown clusters are limited because the computational complex- ity of fitting a model scales quadratically with the number of words in the corpus, or the number of “base clusters” in some efficient implementations, making it infeasible to train it on large corpora or with millions of word types. Although some attempts have been made to train named-entity recognition systems with other forms of word representations, most notably those obtained from training neural language models (Turian et al., 2010; Collobert and Weston, 2008), these systems have historically underperformed simple applications of Brown clusters. A disad- vantage of neural language models is that, while they are inherently more scalable than Brown clus- ters, training large neural networks is still often expensive; for example, Turian et al (2010) re- port that some models took multiple days or weeks to produce acceptable representations. Moreover, language embeddings learned from neural net- works tend to behave in a “nonlinear” fashion, as they are trained to encourage a many-layered neu- ral network to assign high probability to the data. These neural networks can detect nonlinear rela- tionships …

Content truncated to 500 words