Reducing Semantic Drift with Bagging and Distributional Similarity

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 396–404, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Reducing semantic drift with bagging and distributional similarity Tara McIntosh and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {tara,james}@it.usyd.edu.au Abstract Iterative bootstrapping algorithms are typ- ically compared using a single set of hand- picked seeds. However, we demonstrate that performance varies greatly depend- ing on these seeds, and favourable seeds for one algorithm can perform very poorly with others, making comparisons unreli- able. We exploit this wide variation with bagging, sampling from automatically ex- tracted seeds to reduce semantic drift. However, semantic drift still occurs in later iterations. We propose an integrated distributional similarity filter to identify and censor potential semantic drifts, en- suring over 10% higher precision when ex- tracting large semantic lexicons. 1 Introduction Iterative bootstrapping algorithms have been pro- posed to extract semantic lexicons for NLP tasks with limited linguistic resources. Bootstrapping was initially proposed by Riloff and Jones (1999), and has since been successfully applied to extract- ing general semantic lexicons (Riloff and Jones, 1999; Thelen and Riloff, 2002), biomedical enti- ties (Yu and Agichtein, 2003), facts (Pas¸ca et al., 2006), and coreference data (Yang and Su, 2007). Bootstrapping approaches are attractive because they are domain and language independent, re- quire minimal linguistic pre-processing and can be applied to raw text, and are efficient enough for tera-scale extraction (Pas¸ca et al., 2006). Bootstrapping is minimally supervised, as it is initialised with a small number of seed instances of the information to extract. For semantic lexi- cons, these seeds are terms from the category of in- terest. The seeds identify contextual patterns that express a particular semantic category, which in turn recognise new terms (Riloff and Jones, 1999). Unfortunately, semantic drift often occurs when ambiguous or erroneous terms and/or patterns are introduced into and then dominate the iterative process (Curran et al., 2007). Bootstrapping algorithms are typically com- pared using only a single set of hand-picked seeds. We first show that different seeds cause these al- gorithms to generate diverse lexicons which vary greatly in precision. This makes evaluation un- reliable – seeds which perform well on one algo- rithm can perform surprisingly poorly on another. In fact, random gold-standard seeds often outper- form seeds carefully chosen by domain experts. Our second contribution exploits this diversity we have identified. We present an unsupervised bagging algorithm which samples from the ex- tracted lexicon rather than relying on existing gazetteers or hand-selected seeds. Each sample is then fed back as seeds to the bootstrapper and the results combined using voting. This both improves the precision of the lexicon and the robustness of the algorithms to the choice of initial seeds. Unfortunately, semantic drift still dominates in later iterations, since erroneous extracted terms and/or patterns eventually shift the category’s di- rection. Our third contribution focuses on detect- ing and censoring the terms introduced by seman- tic drift. …

Content truncated to 500 words