Improving Open Information Extraction via Iterative Rank-Aware Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5295–5300 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5295 Improving Open Information Extraction via Iterative Rank-Aware Learning Zhengbao Jiang, Pengcheng Yin, Graham Neubig Language Technologies Institute Carnegie Mellon University {zhengbaj, pcyin, gneubig}@cs.cmu.edu Abstract Open information extraction (IE) is the task of extracting open-domain assertions from natu- ral language sentences. A key step in open IE is confidence modeling, ranking the ex- tractions based on their estimated quality to adjust precision and recall of extracted asser- tions. We found that the extraction likelihood, a confidence measure used by current super- vised open IE systems, is not well calibrated when comparing the quality of assertions ex- tracted from different sentences. We propose an additional binary classification loss to cal- ibrate the likelihood to make it more globally comparable, and an iterative learning process, where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. Experiments on OIE2016 demonstrate the effectiveness of our method.1 1 Introduction Open information extraction (IE, Sekine (2006); Banko et al. (2007)) aims to extract open-domain assertions represented in the form of n-tuples (e.g., was born in; Barack Obama; Hawaii) from natural language sentences (e.g., Barack Obama was born in Hawaii). Open IE started from rule- based (Fader et al., 2011) and syntax-driven sys- tems (Mausam et al., 2012; Corro and Gemulla, 2013), and recently has used neural networks for supervised learning (Stanovsky et al., 2018; Cui et al., 2018; Sun et al., 2018; Duh et al., 2017; Jia et al., 2018). A key step in open IE is confidence model- ing, which ranks a list of candidate extractions based on their estimated quality. This is impor- tant for downstream tasks, which rely on trade- offs between the precision and recall of extracted 1Code and data are available at https://github. com/jzbjyb/oie_rank generate model t extractions extractions up to t merge minimize binary classification loss model t+1 extractions up to t+1 Figure 1: Iterative rank-aware learning. assertions. For instance, an open IE-powered medical question answering (QA) system may re- quire its assertions in higher precision (and conse- quently lower recall) than QA systems for other domains. For supervised open IE systems, the confidence score of an assertion is typically com- puted based on its extraction likelihood given by the model (Stanovsky et al., 2018; Sun et al., 2018). However, we observe that this often yields sub-optimal ranking results, with incorrect extrac- tions of one sentence having higher likelihood than correct extractions of another sentence. We hypothesize this is due to the issue of a disconnect between training and test-time objectives. Specif- ically, the system is trained solely to raise like- lihood of gold-standard extractions, and during training the model is not aware of its test-time be- havior of ranking a set of system-generated asser- tions across sentences that potentially include in- correct extractions. To calibrate open IE confidences and …

Content truncated to 500 words