Impact of Corpus Diversity and Complexity on NER Performance

Impact of Corpus Diversity and Complexity on NER Performance Tatyana Shmanina1,2, Ingrid Zukerman1,2, Antonio Jimeno Yepes1,3, Lawrence Cavedon1,3, Karin Verspoor1,3 1NICTA Victoria Research Laboratory, Melbourne, Australia 2Clayton School of Information Technology, Monash University, Australia 3Department of Computing and Information Systems, University of Melbourne, Australia Tatyana.Shmanina@nicta.com.au, Ingrid.Zukerman@monash.edu, Antonio.Jimeno@gmail.com,{Lawrence.Cavedon,Karin.Verspoor}@nicta.com.au Abstract We describe a cross-corpora evaluation of disease mention recognition for two an- notated biomedical corpora: the Human Variome Project Corpus and the Arizona Disease Corpus. Our analysis of the per- formance of a state-of-the-art NER tool in terms of the characteristics and annotation schema of these corpora shows that these factors significantly affect performance. 1 Introduction The recent growth of on-line biomedical literature has spawned an increasing number of NLP tools for content analysis that help researchers and prac- titioners access the latest developments in their fields. Examples of these tools include: BANNER – a Named Entity Recognizer (NER) for the biomed- ical domain (Leaman and Gonzalez, 2008); ABNER – a NER for molecular biology (Settles, 2004); and Whatizit – a Web service which provides func- tionality to perform text-mining tasks (Rebholz- Schuhmann et al., 2008). These tools in turn re- quire the development of annotated training cor- pora, e.g., (Kim et al., 2003; Rosario and Hearst, 2004; Kulick et al., 2004; Pestian et al., 2007; Jimeno-Yepes et al., 2008; Bada et al., 2012). Studies have been conducted to examine the performance of different NLP tools on a single corpus, e.g., (Jacob et al., 2013; Verspoor et al., 2012). However, experience shows that the char- acteristics of a corpus influence performance, e.g., (Cao and Zukerman, 2012) for sentiment analy- sis and (Pyysalo et al., 2008) in the biomedical space. In this paper, we analyze how the char- acteristics and annotation schemas of two corpora influence BANNER’s performance on the recogni- tion of diseases (note that BANNER outperforms AB- NER in the recognition of diseases (Leaman and Gonzalez, 2008)). The corpora in question are the Human Variome Project Corpus (HVPC) devel- oped at NICTA (Verspoor et al., 2013), and the Arizona Disease Corpus (AZDC) – a popular med- ical resource developed at the University of Ari- zona (Leaman et al., 2009).1 Our results show that BANNER’s performance on HVPC significantly exceeds its performance on AZDC. This is (at least partly) explained by differences in corpus characteristics, such as re- duced disease mention diversity resulting from HVPC’s specific focus, and by some requirements of HVPC’s annotation schema. These observations suggest that corpus analysis should be conducted along with performance evaluation in order to ap- propriately assess the obtained results and the suit- ability of a corpus for training general NER tools. 2 Biomedical Corpora AZDC is a biomedical textual resource focusing on disease annotation (Leaman et al., 2009). It was extracted from a corpus created by Craven and Kumlien (1999), which consists of sentences se- lected from MEDLINE R⃝ abstracts via queries for six proteins. All disease mentions in AZDC are an- notated, with each disease annotation containing a Unified Medical Language System R⃝ …

Content truncated to 500 words