Applying Discourse Analysis and Data Mining Methods to Spoken OSCE Assessments

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 577–584 Manchester, August 2008 Applying Discourse Analysis and Data Mining Methods to Spoken OSCE Assessments Meladel Mistica, Timothy Baldwin The University of Melbourne CSSE {mmistica,tim} @csse.unimelb.edu.au Marisa Cordella, Simon Musgrave Monash University School of Languages, Cultures and Linguistics {marisa.cordella,simon.musgrave} @arts.monash.edu.au Abstract This paper looks at the transcribed data of patient-doctor consultations in an exami- nation setting. The doctors are interna- tionally qualified and enrolled in a bridg- ing course as preparation for their Aus- tralian Medical Council examination. In this study, we attempt to ascertain if there are measurable linguistic features of the consultations, and to investigate whether there is any relevant information about the communicative styles of the qualify- ing doctors that may predict satisfactory or non-satisfactory examination outcomes. We have taken a discourse analysis ap- proach in this study, where the core unit of analysis is a ‘turn’. We approach this prob- lem as a binary classification task and em- ploy data mining methods to see whether the application of which to richly anno- tated dialogues can produce a system with an adequate predictive capacity. 1 Introduction This paper describes our experimentation with ap- plying data mining methods to transcribed doctor– patient consultations. It is in essence a discovery project: we apply methods to a field and task that is not ordinarily associated with such approaches in order to ascertain whether this could make for a tractable learning task. The task involves the extraction of dis- course features from doctor–patient consultations performed by international medical graduates c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. (IMGs), and what is known as ‘simulated pa- tients’ (Vu et al., 1994), respectively. The IMGs are enrolled in a bridging course in Melbourne as preparation for their Australian Medical Council (AMC) examination, the successful completion of which is one of the pre-requisites to becoming a fully accredited practitioner in Australia. This par- tially replicates the AMC examination by studying in detail how IMGs perform two objective struc- tured clinical examinations (OSCEs). See Sec- tion 2 for full details of the examination environ- ment and participants involved. The main questions raised when initiating this study were: • How objective is the testing? • What is the importance placed on language skills in OSCE environments? • What makes for a successful OSCE? In this research, we aim to build classifiers that make reasonable predictions of the data being tested, and possibly point us in the right direction with respect to the questions above. From the clas- sifiers we build, we also hope to ascertain which of our features best predict a successful examination. We organise the paper as follows. In Section 2, we briefly describe the examination environment and process, the marking scheme, and the partic- ipants involved in the testing of the IMGs. We also outline some of the issues that have arisen with regard to the current methods of IMG testing. In …

Content truncated to 500 words