Named Entity Recognition with Long Short-Term Memory

Named Entity Recognition with Long Short-Term Memory James Hammerton Alfa-Informatica, University of Groningen Groningen, The Netherlands james@let.rug.nl Abstract In this approach to named entity recognition, a recurrent neural network, known as Long Short-Term Memory, is applied. The network is trained to perform 2 passes on each sentence, outputting its decisions on the second pass. The first pass is used to acquire information for dis- ambiguation during the second pass. SARD- NET, a self-organising map for sequences is used to generate representations for the lexical items presented to the LSTM network, whilst orthogonal representations are used to repre- sent the part of speech and chunk tags. 1 Introduction In this paper, Long Short-Term Mem- ory (LSTM) (Hochreiter and Schmidhuber, 1997) is applied to named entity recognition, using data from the Reuters Corpus, English Language, Volume 1, and the European Corpus Initiative Multilingual Corpus 1. LSTM is an architecture and training algorithm for recur- rent neural networks (RNNs), capable of remembering information over long time periods during the processing of a sequence. LSTM was applied to an earlier CoNLL shared task, namely clause identification (Hammerton, 2001) al- though the performance was significantly below the per- formance of other methods, e.g. LSTM achieved an fs- core of 50.42 on the test data where other systems’ fs- cores ranged from 62.77 to 80.44. However, not all train- ing data was used in training the LSTM networks. Better performance has since been obtained where the complete training set was used (Hammerton, unpublished), yield- ing an fscore of 64.66 on the test data. 2 Representing lexical items An efficient method of representing lexical items is needed. Hammerton (2001; unpublished) employed lex- ical space (Zavrel and Veenstra, 1996) representations of the words which are derived from their co-occurrence statistics. Here, however, a different approach is used. A SARDNET (James and Miikkulainen, 1995), a self- organising map (SOM) for sequences, is trained to form representations of the words and the resulting representa- tions reflect the morphology of the words. James and Miikkulainen (1995) provide a detailed description of how SARDNET operates. Briefly, the SARDNET operates in a similar manner to the standard SOM. It consists of a set of inputs and a set of map units. Each map unit contains a set of weights equal in size to the number of inputs. When an input is presented, the map unit with the closest weights to the input vector is chosen as the winner. When processing a sequence, this winning unit is taken out of the competition for subse- quent inputs. The activation of a winning unit is set at 1 when it is first chosen and then multiplied by a decay factor (here set at 0.9) for subsequent inputs in the se- quence. At the beginning of a new sequence all map units are made available again for the first input. Thus, once a sequence of inputs has been presented, the map units ac- tivated as winners indicate which inputs were presented and the activation levels of those …

Content truncated to 500 words