Combining Generative and Discriminative Model Scores for Distant Supervision

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 24–29, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Combining Generative and Discriminative Model Scores for Distant Supervision Benjamin Roth, Dietrich Klakow Saarland University Spoken Language Systems Saarbr¨ucken, Germany {benjamin.roth|dietrich.klakow}@lsv.uni-saarland.de Abstract Distant supervision is a scheme to generate noisy training data for relation extraction by aligning entities of a knowledge base with text. In this work we combine the output of a discriminative at-least-one learner with that of a generative hierarchical topic model to re- duce the noise in distant supervision data. The combination significantly increases the rank- ing quality of extracted facts and achieves state-of-the-art extraction performance in an end-to-end setting. A simple linear interpo- lation of the model scores performs better than a parameter-free scheme based on non- dominated sorting. 1 Introduction Relation extraction is the task of finding relational facts in unstructured text and putting them into a structured (tabularized) knowledge base. Training machine learning algorithms for relation extraction requires training data. If the set of relations is pre- specified, the training data needs to be labeled with those relations. Manual annotation of training data is laborious and costly, however, the knowledge base may al- ready partially be filled with instances from the rela- tions. This is utilized by a scheme known as distant supervision (DS) (Mintz et al., 2009): text is au- tomatically labeled by aligning (matching) pairs of entities that are contained in a knowledge base with their textual occurrences. Whenever such a match is encountered, the surrounding context (sentence) is assumed to express the relation. This assumption, however, can fail. Consider the example given in (Taka- matsu et al., 2012): If the tuple place_of_birth(Michael Jackson, Gary) is contained in the knowledge base, one matching context could be: Michael Jackson was born in Gary ... And another possible context: Michael Jackson moved from Gary ... Clearly, only the first context indeed expresses the relation and should be labeled accordingly. Three basic approaches have been proposed to deal with noisy distant supervision instances: The discriminative at-least-one approach (Riedel et al., 2010), that requires that at least one of the matches for a relation-entity tuple indeed expresses the relation; The generative approach (Alfonseca et al., 2012) that separates relation-specific distribu- tions from noise distributions by using hierarchical topic models; And the pattern correlation approach (Takamatsu et al., 2012) that assumes that contexts which match argument pairs have a high overlap in argument pairs with other patterns expressing the re- lation. In this work we combine 1) a discriminative at- least-one learner, that requires high scores for both a dedicated noise label and the matched relation, and 2) a generative topic model that uses a feature-based representation to separate relation-specific patterns from background or pair-specific noise. We score surface patterns and show that combining the two approaches results in a better ranking quality of re- lational facts. In an end-to-end evaluation we set a threshold on the pattern scores and apply the …

Content truncated to 500 words