Multi-label Annotation in Scientific Articles - The Multi-label Cancer Risk Assessment Corpus

Multi-label Annotation in Scientific Articles - The Multi-label Cancer Risk Assessment Corpus James Ravenscroft1, Anika Oellrich2, Shyamasree Saha3, Maria Liakata4 1 University of Warwick, Coventry, CV47AL 2 King’s College London, Denmark Hill, London, SE5 8AF, UK 3 Queen Mary University London, Mile End Road, London E1 4NS 4 University of Warwick, Coventry, CV47AL, m.liakata@warwick.ac.uk Abstract With the constant growth of the scientific literature, automated processes to enable access to its contents are increasingly in demand. Several functional discourse annotation schemes have been proposed to facilitate information extraction and summarisation from scientific articles, the most well known being argumentative zoning. Core Scientific concepts (CoreSC) is a three layered fine-grained annotation scheme providing content-based annotations at the sentence level and has been used to index, extract and summarise scientific publications in the biomedical literature. A previously developed CoreSC corpus on which existing automated tools have been trained contains a single annotation for each sentence. However, it is the case that more than one CoreSC concept can appear in the same sentence. Here, we present the Multi-CoreSC CRA corpus, a text corpus specific to the domain of cancer risk assessment (CRA), consisting of 50 full text papers, each of which contains sentences annotated with one or more CoreSCs. The full text papers have been annotated by three biology experts. We present several inter-annotator agreement measures appropriate for multi-label annotation assessment. Employing several inter-annotator agreement measures, we were able to identify the most reliable annotator and we built a harmonised consensus (gold standard) from the three different annotators, while also taking concept priority (as specified in the guidelines) into account. We also show that the new Multi-CoreSC CRA corpus allows us to improve performance in the recognition of CoreSCs. The updated guidelines, the multi-label CoreSC CRA corpus and other relevant, related materials are available at the time of publication at http://www.sapientaproject.com/. Keywords: Multi-label annotations, functional discourse, scientific discourse, Core Scientific Concepts, Cancer Risk Assessment 1. Background 1.1. The CoreSC scheme, corpus and applications In order to extract semantic information from textone needs to pay attention to different aspects of the dis- course such as how different sentences or clauses in- terconnect, alternative mentions of the same entities or concepts, change of theme or topic, communication roles served by different discourse segments (Webber et al., 2012). The largest resource for discourse an- notation remains the Penn Discourse TreeBank which contains over 18k explicitly signalled relations (Prasad et al., 2014; Prasad et al., 2011). Core Scientific Con- cepts (CoreSC)(Liakata et al., 2010; Liakata and others, 2012) is a three-layer functional discourse scheme used to annotate scientific publications at the sentence level. The first layer corresponds to 11 categories (Hypoth- esis, Motivation, Background, Goal, Object, Method, Experiment, Model, Observation, Result, Conclusion), deemed suitable in expressing the structure of a sci- entific investigation while the second layer provides for the annotation of properties of the concepts (e.g. “New”, “Old”). A depiction of the first and second layer in a single flattened representation can be found in Table 1. …

Content truncated to 500 words