Automated Generation of Test Suites for Error Analysis of Concept Recognition Systems

Automated Generation of Test Suites for Error Analysis of Concept Recognition Systems Tudor Groza1,2 1School of ITEE The University of Queensland 2Garvan Institute of Medical Research Australia tudor.groza@uq.edu.au Karin Verspoor Dept of Computing and Information Systems The University of Melbourne Australia karin.verspoor@unimelb.edu.au Abstract We present an architecture and implemen- tation of a system that builds structured test suites for concept recognition systems. The system applies provided test case def- initions to a target concept vocabulary, to generate test cases organised accord- ing to those definitions. Test case defini- tions capture particular characteristics, or produce regular transformations, of con- cept terms. The test suites produced by the system enable detailed, systematic, er- ror analysis of the performance of concept recognition systems. 1 Introduction In this paper, we introduce a framework to auto- mate development of test suites for ontology con- cept recognition systems. The objective of the work is to enable the assessment of system com- petence and performance, by organising test cases into groups based on carefully defined characteris- tics. While failure analysis is often done in terms of such characteristics, it is generally done in an unsystematic manner. By providing a framework for automatically building test suites, we aim to enable more systematic investigation of errors. We focus in this initial work on ontology con- cept recognition systems, that is, systems that aim to detect concepts defined in an ontology in nat- ural language text. Prior work has demonstrated substantial differences in the performance of such systems, due to linguistic variability in the expres- sion of ontology concepts (Funk et al., 2014). The use of structured test suites has been shown to en- able identification of performance errors of such systems (Cohen et al., 2010), as well as being use- ful for finding bugs (Cohen et al., 2008). Struc- tured test suites enable systematic evaluation, ex- haustivity, inclusion of negative data, and control over data (Oepen et al., 1998). They can focus on specific linguistic phenomena, that can be pre- sented in isolation and in controlled combinations. Evaluation of the performance of NLP methods is typically done with respect to annotated training data. Methods are assessed based on their abil- ity to reproduce human performance on a task, as measured in terms of the standard metrics of precision, recall, and F-score. Such metrics pro- vide a quantitative basis for comparing perfor- mance of different methods. However, they are by their nature aggregative, considering all anno- tations in the corpus as equal for evaluation pur- poses. Furthermore, such metrics do not provide insight into the nature of errors made by the meth- ods. As stated by (Cohen et al., 2004), testing a system on an annotated corpus “tells you how of- ten the system failed, but not what it failed at; it tells you how often the system succeeds, but not where its strengths are.” Yet investigation of the strengths and failures of a system can reveal infor- mation meaningful for improving system perfor- mance, and is a critical component …

Content truncated to 500 words