Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations

Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations Diego Marcheggiani ILLC University of Amsterdam marcheggiani@uva.nl Ivan Titov ILLC University of Amsterdam titov@uva.nl Abstract We present a method for unsupervised open- domain relation discovery. In contrast to previous (mostly generative and agglomera- tive clustering) approaches, our model relies on rich contextual features and makes mini- mal independence assumptions. The model is composed of two parts: a feature-rich re- lation extractor, which predicts a semantic relation between two entities, and a factor- ization model, which reconstructs arguments (i.e., the entities) relying on the predicted re- lation. The two components are estimated jointly so as to minimize errors in recovering arguments. We study factorization models in- spired by previous work in relation factoriza- tion and selectional preference modeling. Our models substantially outperform the genera- tive and agglomerative-clustering counterparts and achieve state-of-the-art performance. 1 Introduction The task of Relation Extraction (RE) consists of de- tecting and classifying the semantic relations present in text. RE has been shown to benefit a wide range of NLP tasks, such as information retrieval (Liu et al., 2014), question answering (Ravichandran and Hovy, 2002) and textual entailment (Szpektor et al., 2004). Supervised methods for RE have been success- ful when small restricted sets of relations are con- sidered. However, human annotation is expensive and time-consuming, and consequently these ap- proaches do not scale well to the open-domain set- ting where a large number of relations need to be detected in a heterogeneous text collection (e.g., the entire Web). Though weakly-supervised ap- proaches, such as distantly supervised methods and bootstrapping (Mintz et al., 2009; Agichtein and Gravano, 2000), reduce the amount of necessary su- pervision, they still require examples for every rela- tion considered. These limitations led to the emergence of unsu- pervised approaches for RE. These methods extract surface or syntactic patterns between two entities and either directly use these patterns as substitutes for semantic relations (Banko et al., 2007; Banko and Etzioni, 2008) or cluster the patterns (sometimes in context-sensitive way) to form relations (Lin and Pantel, 2001; Yao et al., 2011; Nakashole et al., 2012; Yao et al., 2012). The existing methods, given their generative (or agglomerative clustering) nature, rely on simpler features than their supervised coun- terparts and also make strong modeling assumptions (e.g., assuming that arguments are conditionally in- dependent of each other given the relation). These shortcomings are likely to harm their performance. In this work, we tackle the aforementioned chal- lenges and introduce a new model for unsupervised relation extraction. We also describe an efficient es- timation algorithm which lets us experiment with large unannotated collections. Our model is com- posed of two components: • an encoding component: a feature-rich relation extractor which predicts a semantic relation be- tween two entities in a specific sentence given contextual features; • a reconstruction component: a factorization model which reconstructs arguments (i.e., the entities) relying on the predicted relation. The two components are estimated jointly so as to minimize errors in reconstructing arguments. While …

Content truncated to 500 words