Instance Sampling Methods for Pronoun Resolution

International Conference RANLP 2009 - Borovets, Bulgaria, pages 478–483 Instance Sampling Methods for Pronoun Resolution Holger Wunsch University of T¨ubingen wunsch@sfs.uni-tuebingen.de Sandra K¨ubler Indiana University skuebler@indiana.edu Rachael Cantrell Indiana University rcantrel@indiana.edu Abstract Instance sampling is a method to balance ex- tremely skewed training sets as they occur, for example, in machine learning settings for anaphora resolution. Here, the number of nega- tive samples (i.e. non-anaphoric pairs) is usually substantially larger than the number of positive samples. This causes classifiers to be biased to- wards negative classification, leading to subopti- mal performance. In this paper, we explore how different tech- niques of instance sampling influence the per- formance of an anaphora resolution system for German given different classifiers. All sampling methods prove to increase the F-score for all clas- sifiers, but the most successful method is ran- dom sampling. In the best setting, the F-score improves from 0.541 to 0.608 for memory-based learning, from 0.561 to 0.611 for decision tree learning and from 0.511 to 0.584 for maximum entropy learning. 1 Introduction Machine learning approaches to anaphora resolution are generally defined as deciding, for a pair consisting of a pronoun and a possible antecedent (markable), whether or not they share an anaphoric relation. In the sentence “When the car hit the tree in the dark, it lost a tire.”, for example, the task is to decide whether the pronoun it refers to one of the noun phrases the car, the tree, the dark, or to any of the noun phrases in the preceding sentences in the text. This means that it is paired with each of these noun phrases individually, and the classifier decides for each case whether there is an anaphoric relationship between the two. Training data is produced in the same way. This results in a highly skewed class distribution with many more neg- ative examples than positive ones. For example, Zhao and Ng [17] report a ratio of positive and negative examples of 1:29 for their Chinese data set. Ng and Cardie [7] report that in the MUC-6 data set, only 2% of the pairs are positive examples, all the others are negative (approximate ratio: 1:48); for MUC-7, there are 3% positive examples (approximate ratio: 1:48.5). Such an extreme skewedness of the data set tends to cause suboptimal performance in machine learning ap- proaches. For this reason, instance sampling is often used in order to create a more balanced training set. In our case, this means restricting the number of negative examples while the positive ones remain unchanged. In the case of coreference resolution for definite noun phrases, in which case all preceding markables of a coreference chain are used for positive examples, sam- pling those positive examples can have a positive ef- fect, too, as Ng and Cardie [7] show. One often-used linguistically motivated approach to restrict the negative examples is to use only the mark- ables between the pronoun and the actual antecedent. (cf. for example [7, 10]) Another possibility is to sam- ple the negative …

Content truncated to 500 words