Evaluation of an Algorithm for the Recognition and Classification of Proper Names

Evaluation of an Algorithm for the Recognition and Classification of Proper Names Takahiro Wakao Robert Gaizauskas Yorick Wilks Department of Computer Science, University of Sheffield {T. Wakao, R. Gaizauskas, Y. Wilks}@dcs. shef. ac. uk Abstract We describe an information extraction sys- tem in which four classes of naming expres- sions organisation, person, location and tinm names are recognised and classified with nearly 92% combined precision and re- call. The system applies a mixture of tech- niques to perform this task and these are described in detail. We have quantitatively evaluated the system against a blind test set of Wall Street Journal business articles and report results not only for the system as a whole, but for each component technique and for each class of name. These results show that in order to have high recall, the system needs to make use not only of in- formation internal to the naming expression but also information from outside the nmne. They also show that the contribution of each system component w~ries fl'om one (:lass of name expression to another. 1 Introduction The appropriate treatment of proper names is es- sential in a natural language understanding sys- tem which processes unedited newswire text, since up to 10 % of this type of text may consist of proper names (Coates-Stephens, 1992). Nor is it only the sheer volume of names that makes them important; for some applications, such as inform- ation extraction (IE), robust handling of proper names is a prerequisite for successflflly performing other tasks such as template filling where correctly identifying the entities which play semantic roles in relational frames is crucial. Recent research in the fifth and sixth Message Understanding Con- ferences (MUC5, 1993) (MUC6, 1995) has shown that the recognition and classification of proper names in business newswire text can now be done on a large scale and with high accuracy: the suc- cess rates of the best systems now approach 96%. We have developed an IE system - LaSIf (Large Scale Information Extraction) (Gaizauslms ct al, 1995) which extracts important facts from tmsiness newswire texts. As a key part of the extraction task, the system recognises and clas- sifies certain types of naming expressions, namely those specified in the MUC-6 named entity (NE) task definition (MUC6, 1995). These include or- ganisation, person, and location names, time ex- pressions, percentage expressions, and monetary amount expressions. As defined for MUC-6, the first three of these are proper names, the fourth contains some expressions that would be classi- fied as proper names by linguists and some that; would not, while the last two would generally not be thought of as proper names. In this paper we concentrate only the behaviour of the LaSIE sys- tem with regards to recognising and classifying ex- pressions in the first four classes, i.e. those which consist entirely or in part of proper names (though nothing hangs on omitting the others). The ver- sion of the system reported here achieves ahnost 92% combilmd precision and recall scores …

Content truncated to 500 words