Prototype-Driven Learning for Sequence Models

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 320–327, New York, June 2006. c⃝2006 Association for Computational Linguistics Prototype-Driven Learning for Sequence Models Aria Haghighi Computer Science Division University of California Berkeley aria42@cs.berkeley.edu Dan Klein Computer Science Division University of California Berkeley klein@cs.berkeley.edu Abstract We investigate prototype-driven learning for pri- marily unsupervised sequence modeling. Prior knowledge is specified declaratively, by provid- ing a few canonical examples of each target an- notation label. This sparse prototype information is then propagated across a corpus using distri- butional similarity features in a log-linear gener- ative model. On part-of-speech induction in En- glish and Chinese, as well as an information extrac- tion task, prototype features provide substantial er- ror rate reductions over competitive baselines and outperform previous work. For example, we can achieve an English part-of-speech tagging accuracy of 80.5% using only three examples of each tag and no dictionary constraints. We also compare to semi-supervised learning and discuss the system’s error trends. 1 Introduction Learning, broadly taken, involves choosing a good model from a large space of possible models. In su- pervised learning, model behavior is primarily de- termined by labeled examples, whose production requires a certain kind of expertise and, typically, a substantial commitment of resources. In unsu- pervised learning, model behavior is largely deter- mined by the structure of the model. Designing models to exhibit a certain target behavior requires another, rare kind of expertise and effort. Unsuper- vised learning, while minimizing the usage of la- beled data, does not necessarily minimize total ef- fort. We therefore consider here how to learn mod- els with the least effort. In particular, we argue for a certain kind of semi-supervised learning, which we call prototype-driven learning. In prototype-driven learning, we specify prototyp- ical examples for each target label or label configu- ration, but do not necessarily label any documents or sentences. For example, when learning a model for Penn treebank-style part-of-speech tagging in En- glish, we may list the 45 target tags and a few exam- ples of each tag (see figure 4 for a concrete prototype list for this task). This manner of specifying prior knowledge about the task has several advantages. First, is it certainly compact (though it remains to be proven that it is effective). Second, it is more or less the minimum one would have to provide to a human annotator in order to specify a new annota- tion task and policy (compare, for example, with the list in figure 2, which suggests an entirely different task). Indeed, prototype lists have been used ped- agogically to summarize tagsets to students (Man- ning and Sch¨utze, 1999). Finally, natural language does exhibit proform and prototype effects (Radford, 1988), which suggests that learning by analogy to prototypes may be effective for language tasks. In this paper, we consider three sequence mod- eling tasks: part-of-speech tagging in English and Chinese and a classified ads information extraction task. Our general approach is to use distributional similarity …

Content truncated to 500 words