Deep Residual Learning for Weakly-Supervised Relation Extraction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1803–1807 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Deep Residual Learning for Weakly-Supervised Relation Extraction Yi Yao Huang Department of Electrical Engineering National Taiwan University Taipei, Taiwan b02901042@ntu.edu.tw William Yang Wang Department of Computer Science University of California, Santa Barbara Santa Barbara, CA 93106 USA william@cs.ucsb.edu Abstract Deep residual learning (ResNet) (He et al., 2016) is a new method for training very deep neural networks using identity map- ping for shortcut connections. ResNet has won the ImageNet ILSVRC 2015 clas- sification task, and achieved state-of-the- art performances in many computer vi- sion tasks. However, the effect of resid- ual learning on noisy natural language pro- cessing tasks is still not well understood. In this paper, we design a novel convolu- tional neural network (CNN) with resid- ual learning, and investigate its impacts on the task of distantly supervised noisy rela- tion extraction. In contradictory to popu- lar beliefs that ResNet only works well for very deep networks, we found that even with 9 layers of CNNs, using identity map- ping could significantly improve the per- formance for distantly-supervised relation extraction. 1 Introduction Relation extraction is the task of predicting at- tributes and relations for entities in a sentence (Ze- lenko et al., 2003; Bunescu and Mooney, 2005; GuoDong et al., 2005). For example, given a sentence “Barack Obama was born in Honolulu, Hawaii.”, a relation classifier aims at predicting the relation of “bornInCity”. Relation extraction is the key component for building relation knowl- edge graphs, and it is of crucial significance to natural language processing applications such as structured search, sentiment analysis, question an- swering, and summarization. A major issue for relation extraction is the lack of labeled training data. In recent years, distant supervision (Mintz et al., 2009; Hoffmann et al., 2011; Surdeanu et al., 2012) emerges as the most popular method for relation extraction— it uses knowledge base facts to select a set of noisy in- stances from unlabeled data. Among all the ma- chine learning approaches for distant supervision, the recently proposed Convolutional Neural Net- works (CNNs) model (Zeng et al., 2014) achieved the state-of-the-art performance. Following their success, Zeng et al. (2015) proposed a piece-wise max-pooling strategy to improve the CNNs. Var- ious attention strategies (Lin et al., 2016; Shen and Huang, 2016) for CNNs are also proposed, obtaining impressive results. However, most of these neural relation extraction models are rela- tively shallow CNNs—typically only one convo- lutional layer and one fully connected layer were involved, and it was not clear whether deeper mod- els could have benefits on distilling signals from noisy inputs in this task. In this paper, we investigate the effects of train- ing deeper CNNs for distantly-supervised relation extraction. More specifically, we designed a con- volutional neural network based on residual learn- ing (He et al., 2016)—we show how one can in- corporate word embeddings and position embed- dings into a deep residual network, while feed- …

Content truncated to 500 words