Composition of Conditional Random Fields for Transfer Learning

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 748–754, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Composition of Conditional Random Fields for Transfer Learning Charles Sutton and Andrew McCallum Department of Computer Science University of Massachusetts Amherst, MA 01003 {sutton,mccallum}@cs.umass.edu Abstract Many learning tasks have subtasks for which much training data exists. Therefore, we want to transfer learning from the old, general- purpose subtask to a more specific new task, for which there is often less data. While work in transfer learning often considers how the old task should affect learning on the new task, in this paper we show that it helps to take into account how the new task affects the old. Specifically, we perform joint decoding of separately-trained sequence models, preserv- ing uncertainty between the tasks and allowing information from the new task to affect predic- tions on the old task. On two standard text data sets, we show that joint decoding outperforms cascaded decoding. 1 Introduction Many tasks in natural language processing are solved by chaining errorful subtasks. Within information extrac- tion, for example, part-of-speech tagging and shallow parsing are often performed before the main extraction task. Commonly these subtasks have their own standard sets of labeled training data: for example, many large data sets exist for learning to extract person names from newswire text; whereas the available training data for new applications, such as extracting appointment information from email, tends to be much smaller. Thus, we need to transfer regularities learned from a well-studied subtask, such as finding person names in newswire text, to a new, related task, such as finding names of speakers in email seminar announcements. In previous NLP systems, transfer is often accom- plished by training a model for the subtask, and using its prediction as a feature for the new task. For example, re- cent CoNLL shared tasks (Tjong Kim Sang & De Meul- der, 2003; Carreras & Marquez, 2004), which are stan- dard data sets for such common NLP tasks as clause iden- tification and named-entity recognition, include predic- tions from a part-of-phrase tagger and a shallow parser as features. But including only the single most likely sub- task prediction fails to exploit useful dependencies be- tween the tasks. First, if the subtask prediction is wrong, the model for the new task may not be able to recover. Of- ten, errors propagate upward through the chain of tasks, causing errors in the final output. This problem can be ameliorated by preserving uncertainty in the subtask pre- dictions, because even if the best subtask prediction is wrong, the distribution over predictions can still be some- what accurate. Second, information from the main task can inform the subtask. This is especially important for learning trans- fer, because the new domain often has different charac- teristics than the old domain, which is often a standard benchmark data set. For example, named-entity recog- nizers are usually trained on newswire text, which is more structured …

Content truncated to 500 words