Using Confusion Graphs to Understand Classifier Error

Proceedings of 2016 NAACL Human-Computer Question Answering Workshop, pages 48–52, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Using Confusion Graphs to Understand Classifier Error Davis Yoshida and Jordan Boyd-Graber davis.yoshida@colorado.edu, jordan.boyd.graber@colorado.edu Abstract Understanding the nature of the errors of a machine learning system is often difficult for multiclass classification problems with a large number of classes. This is true even more so if the number of examples for each class is low. To interpret the performance of a mul- ticlass classifier, we form a graph represent- ing the errors, and use average-link clustering to find groups of classes which are confused with each other. We apply this idea to the QANTA question answering system (Iyyer et al., 2014), and provide a method of analysis of the clusters. 1 Introduction A typical development process for a machine learn- ing pipeline involves many iterations of error anal- ysis, and changes to each step of the pipeline. One of the most common tools for understanding the er- rors of multiclass classifiers is a confusion matrix. If, however, the number of classes is very large rela- tive to the number of samples, the confusion matrix will be both large and sparse. This is problematic for error analysis, because adding a feature which reduces the confusion between two classes amounts to decreasing the overall error by a tiny increment. If the error is attacked on a class by class basis, the process will be both very lengthy and result in fea- tures which may not discriminate between any pairs of classes other than the one they were created for. While it is certainly possible to engineer features which are widely useful, it would be beneficial to do so this both consistently and quickly. One example of a multiclass classification system with a large number of classes is QANTA, a factoid question answering system. QANTA is designed to play the Quiz Bowl trivia game, in which players are given questions word by word, and want to answer the question correctly as early as possible. Since the number of answers QANTA is trained on is finite, it can be viewed as a multiclass classifier with a large number of classes. For any given pair of answers, it is highly unlikely that there are more than a few questions for which the system confused one answer with the other. This makes QANTA a perfect ex- ample of a multiclass classifier which has a sparse confusion matrix. To better analyze the errors com- mitted by the system we form a confusion graph for each of three subsets of our data. The construction of these graphs and some of their interesting proper- ties are discussed in Section 2. In Section 3, we discuss the use of average-link agglomerative clustering to find clusters of answers which are highly confused with each other on a val- idation set. In Section 4, we apply the Autoslog system (Riloff, 1996) to each cluster to find find textual fea- tures which …

Content truncated to 500 words