A Corpus of Clinical Practice Guidelines Annotated with the Importance of Recommendations

A Corpus of Clinical Practice Guidelines Annotated with the Importance of Recommendations Jonathon Read♠ Erik Velldal♡ Marc Cavazza♢ Gersende Georg♣ ♠ Teesside University, School of Computing ♡ University of Oslo, Department of Informatics ♢ University of Kent, School of Engineering and Digital Arts ♣ Haute Autorit´e de Sant´e j.read@tees.ac.uk, erikve@ifi.uio.no, m.o.cavazza@kent.ac.uk, g.georg@has-sante.fr Abstract In this paper we present the Corpus of REcommendation STrength (CREST), a collection of HTML-formatted clinical guidelines annotated with the location of recommendations. Recommendations are labelled with an author-provided indicator of their strength of importance. As data was drawn from many disparate authors, we define a unified scheme of importance labels, and provide a mapping for each guideline. We demonstrate the utility of the corpus and its annotations in some initial measurements investigating the type of language construc- tions associated with strong and weak recommendations, and experiments into promising features for recommendation classification, both with respect to strong and weak labels, and to all labels of the unified scheme. An error analysis indicates that, while there is a strong relationship between lexical choices and strength labels, there can be substantial variance in the choices made by different authors. Keywords: Corpus annotation, Extra-propositional aspects of meaning, Normalised Pointwise Mutual Information, Support Vec- tor Machines 1. Introduction Practice guidelines document various recommendations re- garding best practice for particular clinical situations. They provide assistance to medical practitioners and their pa- tients when making decisions about individual health care (Field and Lohr, 1990), and can also influence high-level health care policy making. Guideline authors often annotate their recommendations with labels that indicate the strength of importance of the statement (see Figure 1 for examples). Such labels present an interesting resource that can add a further dimension to current work in extra-propositional aspects of mean- ing (Morante and Sporleder, 2012; Blanco et al., 2015)— namely the investigation of linguistic constructions that convey the importance of propositions. To support such work we have developed the Corpus of REcommendation STrength (CREST), a collection of clinical guidelines an- notated with instances of recommendations together with their strength of importance as specified by their authors.1 In this paper we introduce related work in Section 2, before describing how we created our corpus in Section 3. Sec- tion 4 presents an initial analysis of some of the features of the strength of importance. Then in Section 5 we present the results and an error analysis of some initial classifica- tion experiments including the discrimination between the two most frequently occurring labels as they multi-class problem involving all labels. We then conclude in Section 6 and outline our plans for further development and investi- gation of the corpus. 1The corpus is freely available to download from http://www.velldal.net/erik/crest/ 2. Related work Zhang and Xu (2015) created a corpus of emails anno- tated with their importance, where participants solicited through Mechanical Turk (rather than the authors) chose labels of important, normal and not important. The authors found that a machine learning-based classifier could iden- tify unimportant emails with a precision …

Content truncated to 500 words