Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1388–1398 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1388 Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction Christoph Alt Marc H¨ubner Leonhard Hennig German Research Center for Artificial Intelligence (DFKI) Speech and Language Technology Lab {christoph.alt, marc.huebner, leonhard.hennig}@dfki.de Abstract Distantly supervised relation extraction is widely used to extract relational facts from text, but suffers from noisy labels. Current re- lation extraction methods try to alleviate the noise by multi-instance learning and by pro- viding supporting linguistic and contextual in- formation to more efficiently guide the relation classification. While achieving state-of-the-art results, we observed these models to be biased towards recognizing a limited set of relations with high precision, while ignoring those in the long tail. To address this gap, we utilize a pre-trained language model, the OpenAI Gen- erative Pre-trained Transformer (GPT) (Rad- ford et al., 2018). The GPT and similar mod- els have been shown to capture semantic and syntactic features, and also a notable amount of “common-sense” knowledge, which we hy- pothesize are important features for recogniz- ing a more diverse set of relations. By extend- ing the GPT to the distantly supervised set- ting, and fine-tuning it on the NYT10 dataset, we show that it predicts a larger set of distinct relation types with high confidence. Manual and automated evaluation of our model shows that it achieves a state-of-the-art AUC score of 0.422 on the NYT10 dataset, and performs es- pecially well at higher recall levels. 1 Introduction Relation extraction (RE), defined as the task of identifying the relationship between concepts mentioned in text, is a key component of many natural language processing applications, such as knowledge base population (Ji and Grishman, 2011) and question answering (Yu et al., 2017). Distant supervision (Mintz et al., 2009; Hoffmann et al., 2011) is a popular approach to heuristically generate labeled data for training RE systems by aligning entity tuples in text with known relation instances from a knowledge base, but suffers from Figure 1: Distant supervision generates noisily labeled relation mentions by aligning entity tuples in a text cor- pus with relation instances from a knowledge base. noisy labels and incomplete knowledge base in- formation (Min et al., 2013; Fan et al., 2014). Fig- ure 1 shows an example of three sentences labeled with an existing KB relation, two of which are false positives and do not actually express the re- lation. Current state-of-the-art RE methods try to ad- dress these challenges by applying multi-instance learning methods (Mintz et al., 2009; Surdeanu et al., 2012; Lin et al., 2016) and guiding the model by explicitly provided semantic and syn- tactic knowledge, e.g. part-of-speech tags (Zeng et al., 2014) and dependency parse informa- tion (Surdeanu et al., 2012; Zhang et al., 2018b). Recent methods also utilize side information, e.g. paraphrases, relation aliases, and entity types (Vashishth et al., 2018). However, we ob- serve that these models are often biased …

Content truncated to 500 words