Stacked Ensembles of Information Extractors for Knowledge-Base Population

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 177–187, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Stacked Ensembles of Information Extractors for Knowledge-Base Population Nazneen Fatema Rajani∗ Vidhoon Viswanathan∗ Yinon Bentor Raymond J. Mooney Department of Computer Science University of Texas at Austin Austin, TX 78712, USA {nrajani,vidhoon,yinon,mooney}@cs.utexas.edu Abstract We present results on using stacking to en- semble multiple systems for the Knowl- edge Base Population English Slot Fill- ing (KBP-ESF) task. In addition to us- ing the output and confidence of each sys- tem as input to the stacked classifier, we also use features capturing how well the systems agree about the provenance of the information they extract. We demon- strate that our stacking approach outper- forms the best system from the 2014 KBP- ESF competition as well as alternative en- sembling methods employed in the 2014 KBP Slot Filler Validation task and several other ensembling baselines. Additionally, we demonstrate that including provenance information further increases the perfor- mance of stacking. 1 Introduction Using ensembles of multiple systems is a stan- dard approach to improving accuracy in machine learning (Dietterich, 2000). Ensembles have been applied to a wide variety of problems in natural language processing, including parsing (Hender- son and Brill, 1999), word sense disambiguation (Pedersen, 2000), and sentiment analysis (White- head and Yaeger, 2010). This paper presents a de- tailed study of ensembling methods for the TAC Knowledge Base Population (KBP) English Slot Filling (ESF) task (Surdeanu, 2013; Surdeanu and Ji, 2014). We demonstrate new state-of-the-art results on this KBP task using stacking (Wolpert, 1992), which trains a final classifier to optimally com- bine the results of multiple systems. We present results for stacking all systems that competed in both the 2013 and 2014 KBP-ESF tracks, training ∗ These authors contributed equally on 2013 data and testing on 2014 data. The re- sulting stacked ensemble outperforms all systems in the 2014 competition, obtaining an F1 of 48.6% compared to 39.5% for the best performing system in the most recent competition. Although the associated KBP Slot Filler Val- idation (SFV) Track (Wang et al., 2013; Yu et al., 2014; Sammons et al., 2014) is officially fo- cused on improving the precision of individual ex- isting systems by filtering their results, frequently participants in this track also combine the results of multiple systems and also report increased re- call through this use of ensembling. However, SFV participants have not employed stacking, and we demonstrate that our stacking approach out- performs existing published SFV ensembling sys- tems. KBP ESF systems must also provide prove- nance information, i.e. each extracted slot-filler must include a pointer to a document passage that supports it (Surdeanu and Ji, 2014). Some SFV systems have used this provenance information to help filter and combine extractions (Sammons et al., 2014). Therefore, we also explored enhancing our stacking approach by including additional in- put features that capture provenance information. By including …

Content truncated to 500 words