SuperCAT: The (New and Improved) Corpus Analysis Toolkit

SuperCAT: The (New and Improved) Corpus Analysis Toolkit K. Bretonnel Cohen1, William A. Baumgartner Jr.1, Irina Temnikova2 1University of Colorado School of Medicine, Aurora, Colorado, USA 2 Qatar Computing Research Institute, Doha, Qatar Abstract This paper reports SuperCAT, a corpus analysis toolkit. It is a radical extension of SubCAT, the Sublanguage Corpus Analysis Toolkit, from sublanguage analysis to corpus analysis in general. The idea behind SuperCAT is that representative corpora have no tendency towards closure—that is, they tend towards infinity. In contrast, non-representative corpora have a tendency towards closure—roughly, finiteness. SuperCAT focuses on general techniques for the quantitative description of the characteristics of any corpus (or other language sample), particularly concerning the characteristics of lexical distributions. Additionally, SuperCAT features a complete re-engineering of the previous SubCAT architecture. Keywords: corpus, representativeness, sublanguage, toolkit 1. Introduction This paper reports SuperCAT, a corpus analysis toolkit. It is a radical extension of SubCAT, the Sublanguage Cor- pus Analysis Toolkit (Temnikova et al., 2014), from sub- language analysis to corpus analysis in general. SubCAT was originally developed to assess the fit of corpora to the sublanguage model. To this end, SubCAT contained Python scripts and applications for analyzing lexical, mor- phological, and sentence type closure (McEnery and Wil- son, 2001), as well as over-represented words and syntactic deviance in corpora. The idea, taken from (McEnery and Wilson, 2001), is that sublanguages exhibit a tendency to- wards closure (roughly finiteness), and often syntactic de- viance. The over-represented-word functionality aids in in- vestigating the lexical/semantic characteristics of the do- main represented in the sublanguage. SubCAT was tested on several different corpora, containing texts from different domains (Temnikova et al., 2013a; Temnikova and Cohen, 2013) and languages (Temnikova et al., 2013b). SuperCAT (Version 2.0 of SubCAT) represents a considerable exten- sion of SubCAT to analyze arbitrary corpora. It quickly became evident that while SubCAT was useful for recog- nizing sublanguages, it also had applications to what might be thought of as the “opposite” task—recognizing when a sample of language has the characteristics of a representa- tive sample of language. Therefore, in the current version, we focus less on sublanguage-specific issues and more on general techniques for quantitative description of the char- acteristics of any corpus (or other language sample). Ad- ditionally, SuperCAT features a complete re-engineering of the SubCAT architecture. This version is distributed via a Java JAR file, allowing the associated programs to be run without requiring installation of Python. A number of R scripts for graphing the results of the Java application anal- yses are also provided. 2. Motivation for SubCAT and SuperCAT The original motivation for SubCAT was to provide tools for evaluating the fit of a sample of language to the sublan- guage model. The sublanguage model describes language in combinations of specific domains and genres (e.g. molec- ular biology scientific journal articles and patents written in English, or patient records written in Bulgarian) (Tem- nikova et al., 2014). Examples of sublanguages that have been identified to date include immunology journal arti- cles (Harris …

Content truncated to 500 words