Unsupervised Pre-training With Seq2Seq Reconstruction Loss for Deep Relation Extraction Models

Unsupervised Pre-training With Sequence Reconstruction Loss for Deep Relation Extraction Models Zhuang Li1, Lizhen Qu1,2, Qiongkai Xu1,2, Mark Johnson3 1 DATA61, Australia 2 The Australian National University 3 Macquarie University lizhuang144@gmail.com, {Lizhen.Qu,Qiongkai.Xu}@data61.csiro.au mark.johnson@mq.edu.au Abstract Relation extraction models based on deep learning have been attracting a lot of at- tention recently. Little research is carried out to reduce their need of labeled training data. In this work, we propose an unsu- pervised pre-training method based on the sequence-to-sequence model for deep re- lation extraction models. The pre-trained models need only half or even less training data to achieve equivalent performance as the same models without pre-training. 1 Introduction Relation extraction (RE) is the task of detect- ing and categorizing semantic relations between named entities mentioned in a text corpus. This is important for a wide variety of practical appli- cations. For example, tourism planning bodies are interested in mining social media such as tweets to identifying which restaurants tourists eat in and which hotels those same tourists stay in. RE has been intensively studied for several years (Chan and Roth, 2011; Chan and Roth, 2010). Recently, RE models based on deep neu- ral networks (DNN) have achieved better perfor- mance than conventional RE models that rely on handcrafted features (Xu et al., 2015). However, these DNN models require a large amount of an- notated training data, which is difficult and ex- pensive to obtain. The data problem is not com- pletely solved by relying on methods such as large external knowledge bases and distant supervision because i) models employing only large knowl- edge bases often still perform poorly on RE (An- geli et al., 2014); ii) the external knowledge bases are incomplete; and iii) many important appli- cations lack the relevant domain specific knowl- edge bases. This paper asks the question: can we use unlabeled data to help training DNN RE models? Although unsupervised pre-training is known to be effective for training deep neural net- works, it remains unclear how to apply it to the recently proposed DNN RE models. The main advantage of deep models (compared to the shal- low counterparts) is that they automatically learn distributed representations of the relevant compo- nents of the model (e.g., words, entities, relations, etc.). If we can encode rich syntactic-semantic patterns of relation expressions into the automat- ically learned, low-dimensional representations, and require these representations to be similar if they play a similar role using only unlabeled data, it should be possible for a DNN RE system to achieve a high level of generalization from only small amount of labeled data. In a relational expression, the named entities and words around it provide useful context in- formation. For example, in the sentence ”By 1982 the BL Cars Ltd division renamed itself Austin Rover Group shortly before the launch of the Maestro.” renamed itself is used much less often than an expression such as was founded in to indicate the relation org:foundedIn. Thus it is likely that was founded in will be found in …

Content truncated to 500 words