Evaluation of Generic Relation Identification

Generic Relation Identification: Models and Evaluation Ben Hachey Centre for Language Technology Capital Markets CRC Limited Macquarie University GPO Box 970 NSW 2109 Australia Sydney NSW 2001 bhachey@cmcrc.com Abstract Generic relation identification (GRI) aims to build models of relation-forming entity pairs that can be transferred across domains without modification of model parameters. GRI has high utility in terms of cheap com- ponents for applications like summarisa- tion, automated data exploration and ini- tialisation of bootstrapping of relation ex- traction. A detailed evaluation of GRI is presented for the first time, including ex- plicit tests of portability between newswire and biomedical domains. Experimental re- sults show that a novel approach incorpo- rating dependency parsing is better in terms of recall. And, accuracy is shown to be comparable across domains. 1 Introduction Relation extraction (RE) aims to identify men- tions of relations in text. A relation mention is de- fined as a predicate ranging over two arguments, where an argument represents concepts, objects or people in the world and the predicate describes the type of stative association or interaction that holds between the things represented by the ar- guments. Input to the RE task consists of source documents with entity mention markup (e.g., Fig- ure 1). The output is a list of relation-forming entity mention pairs and a label indicating the type of relation that holds between them (e.g., Ta- ble 1). This paper addresses the relation iden- tification task, which identifies pairs of relation- forming entity mentions (e.g., “David Murray” and “Amidu Berry” in the example). [place American] saxophonist [person David Murray] recruited [person Amidu Berry] and DJ [person Awadi] from [organisation PBS]. Figure 1: Example input to GRI task (from ACE 2004). Square brackets indicate the extent of entity mentions with type as italicised superscript. Entity 1 Entity 2 Relation Type American David Murray CITIZEN/RESIDENT David Murray Amidu Berry BUSINESS David Murray Awadi BUSINESS Amidu Berry PBS MEMBER-OF-GROUP Awadi PBS MEMBER-OF-GROUP Table 1: Example output from GRI task. Relation types are not part of the relation identification task but are given here for purposes of illustration. Relation extraction (RE) can be addressed us- ing supervised (Zelenko et al., 2005; Blitzer et al., 2006), bootstrapping (Brin, 1998; Riloff and Jones, 1999; Agichtein and Gravano, 2000; Has- san et al., 2006) or generic approaches (Conrad and Utt, 1994; Hasegawa et al., 2004). One way to characterise these different approaches is in terms of adaptation cost, i.e. the amount of work necessary to adapt them to a new domain or task. In these terms, supervised approaches (including rule engineering and supervised machine learn- ing) incur the highest cost as systems need to be built largely from scratch for each new domain. Bootstrapping approaches incur less cost as they require only a small amount of seed data. And generic approaches provide domain adaptation for free as parameters do not need to be modified for new domains or tasks. Another way to char- acterise these approaches is in terms of the ontol- ogy creation problems …

Content truncated to 500 words