Combining Supervised and Unsupervised Enembles for Knowledge Base Population

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1943–1948, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Combining Supervised and Unsupervised Ensembles for Knowledge Base Population Nazneen Fatema Rajani and Raymond J. Mooney Department Of Computer Science The University of Texas at Austin nrajani@cs.utexas.edu, mooney@cs.utexas.edu Abstract We propose an algorithm that combines su- pervised and unsupervised methods to ensem- ble multiple systems for two popular Knowl- edge Base Population (KBP) tasks, Cold Start Slot Filling (CSSF) and Tri-lingual Entity Dis- covery and Linking (TEDL). We demonstrate that it outperforms the best system for both tasks in the 2015 competition, several ensem- bling baselines, as well as a state-of-the-art stacking approach. The success of our tech- nique on two different and challenging prob- lems demonstrates the power and generality of our combined approach to ensembling. 1 Introduction Ensembling multiple systems is a well known stan- dard approach to improving accuracy in several ma- chine learning applications (Dietterich, 2000). En- sembles have been applied to parsing (Henderson and Brill, 1999), word sense disambiguation (Ped- ersen, 2000), sentiment analysis (Whitehead and Yaeger, 2010) and information extraction (IE) (Flo- rian et al., 2003; McClosky et al., 2012). Recently, using stacking (Wolpert, 1992) to ensemble sys- tems was shown to give state-of-the-art results on slot-filling and entity linking for Knowledge Base Population (KBP) (Viswanathan et al., 2015; Ra- jani and Mooney, 2016). Stacking uses supervised learning to train a meta-classifier to combine multi- ple system outputs; therefore, it requires historical data on the performance of each system. Rajani and Mooney (2016) use data from the 2014 iteration of the KBP competition for training and then test on the data from the 2015 competition, therefore they can only ensemble the shared systems that participated in both years. However, we would sometimes like to ensem- ble systems for which we have no historical perfor- mance data. For example, due to privacy, some com- panies may be unwilling to share their performance on arbitrary training sets. Simple methods such as voting permit “unsupervised” ensembling, and sev- eral more sophisticated methods have also been de- veloped for this scenario (Wang et al., 2013). How- ever, such methods fail to exploit supervision for those systems for which we do have training data. Therefore, we present an approach that utilizes su- pervised and unsupervised ensembling to exploit the advantages of both. We first use unsupervised en- sembling to combine systems without training data, and then use stacking to combine this ensembled system with other systems with available training data. Using this new approach, we demonstrate new state-of-the-art results on two NIST KBP challenge tasks: Cold Start Slot-Filling (CSSF)1 and the Tri- lingual Entity Discovery and Linking (TEDL) (Ji et al., 2015). Our approach outperforms the best system as well as other state-of-the-art ensembling methods on both tasks in the most recent 2015 com- petition. There is one previous work on ensembling supervised and unsupervised models using graph- based consensus maximization (Gao et al., 2009), …

Content truncated to 500 words