Unsupervised Information Extraction: Regularizing Discriminative Approaches with Relation Distribution Losses

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1378–1387 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1378 Unsupervised Information Extraction: Regularizing Discriminative Approaches with Relation Distribution Losses Étienne Simon and Vincent Guigue and Benjamin Piwowarski Sorbonne Université, CNRS, Laboratoire d’Informatique de Paris 6 LIP6, F-75005 Paris, France {etienne.simon, vincent.guigue, benjamin.piwowarski}@lip6.fr Abstract Unsupervised relation extraction aims at ex- tracting relations between entities in text. Pre- vious unsupervised approaches are either gen- erative or discriminative. In a supervised setting, discriminative approaches, such as deep neural network classifiers, have demon- strated substantial improvement. However, these models are hard to train without supervi- sion, and the currently proposed solutions are unstable. To overcome this limitation, we in- troduce a skewness loss which encourages the classifier to predict a relation with confidence given a sentence, and a distribution distance loss enforcing that all relations are predicted in average. These losses improve the perfor- mance of discriminative based models, and en- able us to train deep neural networks satisfac- torily, surpassing current state of the art on three different datasets. 1 Introduction Information extraction models aim at discovering the underlying semantic structure linking entities mentioned in a text. This can be used to build knowledge bases, which are widely used in sev- eral applications such as question answering (Yih et al., 2015; Berant et al., 2013), document re- trieval (Dalton et al., 2014) and logical reasoning (Socher et al., 2013). In the relation extraction (RE) task, we are in- terested in discovering the semantic (binary) re- lation that holds between two entities mentioned in text. The end goal is to extract triplets of the form (subject, relation, object). A consider- able amount of work has been conducted on su- pervised or weakly-supervised relation extraction (Kambhatla, 2004; Zeng et al., 2015; Lin et al., 2016), with recent state-of-the-art models using deep neural networks (NN). Developing unsupervised relation extraction models is interesting for three reasons: they (1) do not necessitate labeled data except for vali- dating the models; (2) can uncover new relation types; and (3) can be trained from large unlabeled datasets, and then fine-tuned for specific relations. The first unsupervised models used a cluster- ing (Hasegawa et al., 2004; Banko et al., 2007) or generative (Yao et al., 2011, 2012) approach. The latter, which obtained state-of-the-art perfor- mance, still makes a lot of simplifying hypotheses, such as assuming that the entities are condition- ally independent between themselves given the re- lation. To train more expressive models, a shift to discriminative approaches was necessary. The open question then becomes how to provide a suf- ficient learning signal to the classifier. To the best of our knowledge, only Marcheggiani and Titov (2016) followed this path by leveraging represen- tation learning for modeling knowledge bases, and proposed to use an auto-encoder model: their en- coder extracts the relation from a sentence, that the decoder uses to predict a missing entity. However, their encoder is still limited compared to …

Content truncated to 500 words