Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2670–2680 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Fast and Accurate Entity Recognition with Iterated Dilated Convolutions Emma Strubell Patrick Verga David Belanger College of Information and Computer SciencesAndrew McCallum University of Massachusetts Amherst {strubell, pat, belanger, mccallum}@cs.umass.edu Abstract Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs. Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining per- token vector representations serving as in- put to labeling tasks such as NER (often followed by prediction in a linear-chain CRF). Though expressive and accurate, these models fail to fully exploit GPU par- allelism, limiting their computational ef- ficiency. This paper proposes a faster al- ternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in paral- lel across entire documents. We describe a distinct combination of network struc- ture, parameter sharing and training pro- cedures that enable dramatic 14-20x test- time speedups while retaining accuracy comparable to the Bi-LSTM-CRF. More- over, ID-CNNs trained to aggregate con- text from the entire document are even more accurate while maintaining 8x faster test time speeds. 1 Introduction In order to democratize large-scale NLP and in- formation extraction while minimizing our en- vironmental footprint, we require fast, resource- efficient methods for sequence tagging tasks such as part-of-speech tagging and named entity recog- nition (NER). Speed is not sufficient of course: they must also be expressive enough to tolerate the tremendous lexical variation in input data. The massively parallel computation facilitated by GPU hardware has led to a surge of success- ful neural network architectures for sequence la- beling (Ling et al., 2015; Ma and Hovy, 2016; Chiu and Nichols, 2016; Lample et al., 2016). While these models are expressive and accurate, they fail to fully exploit the parallelism opportu- nities of a GPU, and thus their speed is limited. Specifically, they employ either recurrent neural networks (RNNs) for feature extraction, or Viterbi inference in a structured output model, both of which require sequential computation across the length of the input. Instead, parallelized runtime independent of the length of the sequence saves time and energy costs, maximizing GPU resource usage and min- imizing the amount of time it takes to train and evaluate models. Convolutional neural networks (CNNs) provide exactly this property (Kim, 2014; Kalchbrenner et al., 2014). Rather than compos- ing representations incrementally over each token in a sequence, they apply filters in parallel across the entire sequence at once. Their computational cost grows with the number of layers, but not the input size, up to the memory and threading limita- tions of the hardware. …

Content truncated to 500 words