Eliciting Knowledge from Experts: Automatic Transcript Parsing for Cognitive Task Analysis

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4280–4291 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4280 Eliciting Knowledge from Experts: Automatic Transcript Parsing for Cognitive Task Analysis Junyi Du1, He Jiang1, Jiaming Shen2, Xiang Ren1 1Department of Computer Science, University of Southern California 2Department of Computer Science, University of Illinois at Urbana-Champaign {junyidu, jian567, xiangren}@usc.edu, js2@illinois.edu Abstract Cognitive task analysis (CTA) is a type of analysis in applied psychology aimed at elic- iting and representing the knowledge and thought processes of domain experts. In CTA, often heavy human labor is involved to parse the interview transcript into structured knowl- edge (e.g., flowchart for different actions). To reduce human efforts and scale the pro- cess, automated CTA transcript parsing is de- sirable. However, this task has unique chal- lenges as (1) it requires the understanding of long-range context information in conversa- tional text; and (2) the amount of labeled data is limited and indirect—i.e., context-aware, noisy, and low-resource. In this paper, we pro- pose a weakly-supervised information extrac- tion framework for automated CTA transcript parsing. We partition the parsing process into a sequence labeling task and a text span-pair relation extraction task, with distant supervi- sion from human-curated protocol files. To model long-range context information for ex- tracting sentence relations, neighbor sentences are involved as a part of input. Different types of models for capturing context dependency are then applied. We manually annotate real- world CTA transcripts to facilitate the evalua- tion of the parsing tasks1. 1 Introduction Cognitive task analysis (CTA) is a powerful tool for training, instructional design, and develop- ment of expert systems (Woods et al., 1989; Clark and Estes, 1996) focusing on yielding the knowl- edge and thought processes from domain experts (Schraagen et al., 2000). Traditional CTA methods require interviews with domain experts and pars- ing the interview transcript (transcript) into struc- tured text describing processes (protocol, shown in Fig. 1). However, parsing transcripts requires 1Code is available at: https://github.com/cnrpman/procedural-extraction Figure 1: An example of CTA interview transcript and the human parsed structured text (protocol). In the protocol, splitting by the highlighted line numbers indicating the sources in transcript, phrases in proto- col (called protocol phrases) are abstractive descrip- tion of actions in the transcript. In the transcript, the highlighted numbers are line numbers, and the bolded are text spans matched by protocol phrases. The high- lighted line numbers are provided by human parsing which provide constraint on mapping protocol phrases back to the transcript, but they are noisy and pointing back to a large scope of sentences, instead of the text span we want to extract. heavy human labor, which becomes the major hurdle of scaling up CTA. Therefore, automated approaches to extract structured knowledge from CTA interview transcripts are important for expert systems using massive procedural data. A natural realization of automated CTA is to ap- ply relation extraction (RE) models to parse in- terview text. However, the key challenge here is …

Content truncated to 500 words