Name Tagging with Word Clusters and Discriminative Training

Name Tagging with Word Clusters and Discriminative Training Scott Miller, Jethran Guinness, Alex Zamanian BBN Technologies 10 Moulton Street Cambridge, MA 02138 szmiller@bbn.com Abstract We present a technique for augmenting annotated training data with hierarchical word clusters that are automatically derived from a large unannotated corpus. Cluster membership is encoded in features that are incorporated in a discriminatively trained tagging model. Active learning is used to select training examples. We evaluate the technique for named-entity tagging. Compared with a state-of-the-art HMM-based name finder, the presented technique requires only 13% as much annotated data to achieve the same level of performance. Given a large annotated training set of 1,000,000 words, the technique achieves a 25% reduction in error over the state-of-the-art HMM trained on the same material. 1 Introduction At a recent meeting, we presented name-tagging technology to a potential user. The technology had performed well in formal evaluations, had been applied successfully by several research groups, and required only annotated training examples to configure for new name classes. Nevertheless, it did not meet the user's needs. To achieve reasonable performance, the HMM-based technology we presented required roughly 150,000 words of annotated examples, and over a million words to achieve peak accuracy. Given a typical annotation rate of 5,000 words per hour, we estimated that setting up a name finder for a new problem would take four person days of annotation work – a period we considered reasonable. However, this user's problems were too dynamic for that much setup time. To be useful, the system would have to be trainable in minutes or hours, not days or weeks. We left the meeting thinking about ways to reduce training requirements to no more than a few hours. It seemed that three existing ideas could be combined in a way that might reduce training requirements sufficiently to achieve the objective. First were techniques for producing word clusters from large unannotated corpora (Brown et al., 1990; Pereira et al., 1993; Lee and Pereira, 1999). The resulting clusters appeared to contain a great deal of implicit semantic information. This implicit information, we believed, could serve to augment a small amount of annotated data. Particularly promising were techniques for producing hierarchical clusters at various scales, from small and highly specific to large and more general. To benefit from such information, however, we would need an automatic learning mechanism that could effectively exploit it. Fortunately, a second line of recent research provided a potential solution. Recent work in discriminative methods (Lafferty et al., 2001; Sha and Pereira, 2003, Collins 2002) suggested a framework for exploiting large numbers of arbitrary input features. These methods seemed to have exactly the right c h a r a c t e r i s t i c s for i n cor p or a t i n g t h e statistically-correlated hierarchical word clusters we wished to exploit. Combining these two methods, we suspected, would be sufficient to drastically reduce the number of annotated examples required. However, we also …

Content truncated to 500 words