Counter-Training in Discovery of Semantic Patterns

Counter-Training in Discovery of Semantic Patterns Roman Yangarber Courant Institute of Mathematical Sciences New York University roman@cs.nyu.edu Abstract This paper presents a method for unsu- pervised discovery of semantic patterns. Semantic patterns are useful for a vari- ety of text understanding tasks, in par- ticular for locating events in text for in- formation extraction. The method builds upon previously described approaches to iterative unsupervised pattern acquisition. One common characteristic of prior ap- proaches is that the output of the algorithm is a continuous stream of patterns, with gradually degrading precision. Our method differs from the previous pat- tern acquisition algorithms in that it intro- duces competition among several scenar- ios simultaneously. This provides natu- ral stopping criteria for the unsupervised learners, while maintaining good preci- sion levels at termination. We discuss the results of experiments with several scenar- ios, and examine different aspects of the new procedure. 1 Introduction The work described in this paper is motivated by research into automatic pattern acquisition. Pat- tern acquisition is considered important for a variety of “text understanding” tasks, though our particular reference will be to Information Extraction (IE). In IE, the objective is to search through text for enti- ties and events of a particular kind—corresponding to the user’s interest. Many current systems achieve this by pattern matching. The problem of recall, or coverage, in IE can then be restated to a large ex- tent as a problem of acquiring a comprehensive set of good patterns which are relevant to the scenario of interest, i.e., which describe events occurring in this scenario. Among the approaches to pattern acquisition recently proposed, unsupervised methods1 have gained some popularity, due to the substantial re- duction in amount of manual labor they require. We build upon these approaches for learning IE patterns. The focus of this paper is on the problem of con- vergence in unsupervised methods. As with a variety of related iterative, unsupervised methods, the out- put of the system is a stream of patterns, in which the quality is high initially, but then gradually de- grades. This degradation is inherent in the trade-off, or tension, in the scoring metrics: between trying to achieve higher recall vs. higher precision. Thus, when the learning algorithm is applied against a ref- erence corpus, the result is a ranked list of patterns, and going down the list produces a curve which trades off precision for recall. Simply put, the unsupervised algorithm does not know when to stop learning. In the absence of a good stopping criterion, the resulting list of patterns must be manually reviewed by a human; otherwise one can set ad-hoc thresholds, e.g., on the number of allowed iterations, as in (Riloff and Jones, 1999), or else to resort to supervised training to determine such thresholds—which is unsatisfactory when our 1As described in, e.g., (Riloff, 1996; Riloff and Jones, 1999; Yangarber et al., 2000). goal from the outset is to try to limit supervision. Thus, the lack of natural stopping criteria renders these algorithms less …

Content truncated to 500 words