Annotating Named Entities in Consumer Health Questions

Annotating Named Entities in Consumer Health Questions Halil Kilicoglu∗, Asma Ben Abacha∗, Yassine Mrabet∗, Kirk Roberts† Laritza Rodriguez∗, Sonya E. Shooshan∗, Dina Demner-Fushman∗ ∗Lister Hill National Center for Biomedical Communications National Library of Medicine, National Institutes of Health, Bethesda, MD, USA {kilicogluh,mrabety,ddemner}@mail.nih.gov {asma.benabacha,laritza.rodriguez,sonya.shooshan}@nih.gov †University of Texas Health Science Center at Houston, Houston, TX, USA kirk.roberts@uth.tmc.edu Abstract We describe a corpus of consumer health questions annotated with named entities. The corpus consists of 1548 de-identified questions about diseases and drugs, written in English. We defined 15 broad categories of biomedical named entities for annotation. A pilot annotation phase in which a small portion of the corpus was double-annotated by four annotators was followed by a main phase in which double annotation was carried out by six annotators, and a reconciliation phase in which all annotations were reconciled by an expert. We conducted the annotation in two modes, manual and assisted, to assess the effect of automatic pre-annotation and calculated inter-annotator agreement. We obtained moderate inter-annotator agreement; assisted annotation yielded slightly better agreement and fewer missed annotations than manual annotation. Due to complex nature of biomedical entities, we paid particular attention to nested entities for which we obtained slightly lower inter-annotator agreement, confirming that annotating nested entities is somewhat more challenging. To our knowledge, the corpus is the first of its kind for consumer health text and is publicly available. Keywords: consumer health questions, biomedical named entities, assisted annotation, nested entities 1. Introduction The general public is increasingly turning to online re- sources for health information needs (Tustin, 2010). The National Library of Medicine (NLM) receives health- related questions from a wide range of consumers world- wide. Most health-related questions are concerned with disease information, such as diagnosis, treatment, and prog- nosis, and drug information, including ingredients, generic names, and adverse effects. In 2014, NLM received more than 2,500 questions that were classified as disease-related and more than 2,100 questions that were categorized as drug-related by the customer service staff. We have been building a system to assist customer service staff in answer- ing such questions. Focusing on disease questions only, we previously reported an end-to-end question understanding system that extracts question frames from questions (Kil- icoglu et al., 2013), which form the basis for search en- gine queries. Our previous work in question understanding also involved more intermediate tasks, such as question de- composition and focus recognition (Roberts et al., 2014b), question type recognition (Roberts et al., 2014a), anaphora and ellipsis resolution (Kilicoglu et al., 2013), and spelling correction (Kilicoglu et al., 2015). Named entity recognition and normalization is a core as- pect of most of these tasks (e.g., frame extraction and fo- cus recognition). So far, we have used relatively sim- ple dictionary lookup methods to identify named entities in questions and normalize them to UMLS Metathesaurus concepts (Lindberg et al., 1993). However, it has be- come increasingly clear that more sophisticated methods are needed, since the methods we explored assume well- written questions, whereas consumer health questions are rife …

Content truncated to 500 words