Sources of Performance in CRF Transfer Training: a Business Name-tagging Case Study

International Conference RANLP 2009 - Borovets, Bulgaria, pages 465–470 Sources of Performance in CRF Transfer Training: a Business Name-tagging Case Study Marc Vilain The MITRE Corporation 202 Burlington Rd. Bedford, MA 01730 USA mbv@mitre.org Jonathan Huggins The MITRE Corporation 202 Burlington Rd. Bedford, MA 01730 USA jhuggins@mitre.org Ben Wellner The MITRE Corporation 202 Burlington Rd. Bedford, MA 01730 USA wellner@mitre,org Abstract This paper explores methods for increasing performance of CRF models, with a particular concern for transfer learning. We consider in particular the transfer case from political news to hard-to-tag business news, and show the effectiveness of several methods, including a novel semi-supervised approach. Keywords Entity extraction, machine learning, business intelligence. 1. Introduction: name tagging Named entity recognition is one of the most widely studied problems in computational language processing. It was one of the first tasks to be treated with the corpus-based method, and has remained a touchstone for benchmarking corpus-based algorithms and learning regimens. Part of the enduring interest is that name tagging continues to provide technical challenges that help drive research. In particular, while the fundamentals of training a name tagger are well understood, such barriers to practical application as robust coverage and transfer training remain active research areas. Indeed, name-tagging systems tend to perform best when both training and test data are drawn from the same distribution of sources and sample times. However, even seemingly small divergences between training and test can lead to steep drop-offs in performance. Overcoming this lack of carry-over from training to test is thus a key pre- condition for practical entity recognition applications. This paper addresses this issue in the context of train- ing conditional random fields (CRFs) to tag named entities in business texts. We explore several orthogonal strategies for bringing a name tagger to bear on a new domain, with the aim of providing high test-time performance, robustness to out-of-training phenomena, and minimal transfer training costs. We apply these strategies to a business news corpus, and achieve substantial performance gains while only re- quiring modest investments in transfer training. 2. Tagging business news The potential divergence between a name tagger’s training and test performance was documented as far back as the MUC7 evaluation [15]. At issue was a shift in topic be- tween training and test conditions: from air incidents to satellite launches. While the training and test data were otherwise comparable (same sources, same broad topic of aerospace), several system developers implicated this as a cause for poor test-time performance. In a recent study [18], we sought to quantify this di- vergence in the case of business texts. Our study found a substantial training-to-test performance gap for several mature recent systems trained (or hand-configured) to process current events news. While many of the systems did well with current events, their F scores dropped by 15 to 25 points for business news and financial reports. The present paper takes these observations as a chal- lenge to train a business entity tagger. The business genre is primarily of interest here …

Content truncated to 500 words