An Attempt to Use Weighted Cusums to Identify Sublanguages

II II II II II II II II II II II II II II II il II An Attempt to Use Weighted Cusums to Identify Sublanguages Harold Somers Centre for Computational Linguistics, UMIST PO Box 88, Manchester M60 1QD England harol d@ccl, umist, ac. uk Abstract This paper explores the use of weighted cusums, a technique found in authorship attribution studies, for the purpose of identifying sublanguages. The tech- nique, and its relation to standard cusums (cumulative sum charts) is first described, and the formulae for calculations given in detail. The technique compares texts by testing for the incidence of linguistic 'features' of a superficial nature, e.g. proportion of 2- and 3---letter words, words beginning with a vowel, and. so on, and measures whether two texts differ significantly in respect of these features. The paper describes an experiment in which 14 groups of three texts each repre- senting different sublanguages are compared with each other using the technique. The texts are first compared within each group to establish that the technique can identify the groups as being homogeneous. The texts are then compared with each other, and the results analysed. Taking the average of seven different tests, the technique is able to distinguish the sublanguages in only 43% of the case. But if the best score is taken, 79% of pairings can be distinguished. This is a better result, and the test seems able to quantify the difference between sublanguages. Keywords: sublanguage, genre, register, weighted cusum. 1 Introduction This paper concerns a technique which we use to measure whether two texts are representative of the same text genre or sublanguage. It is very much in the spirit of the well-known work in this field by Douglas Biber (1988, 1990, 1995), but differs crucially in that we avoid the explicit selection of linguistic features thought a priori likely to be important in distinguishing sublangnages, and instead use a set of low-level features based on trivial aspects of the words such as length and initial letter. Our technique is borrowed from the neighbouring field of authorship attribution (for an overview of this field see Ule 1982; Smith 1982; Potter 1991; Burrows 1992; Holmes 1994). It is a straightforward calculation, simple to implement, and very general in application. It can be used with fairly small texts. This paper describes an experiment to see whether the technique can be used for the sublanguage identification task, even though it was originally designed for a somewhat different problem. In Somers (forthcoming), we used a technique called 'weighted cusums' to investigate how well a parody of Lewis Carroll had imitated his style. Look- hag also at other writings by Carroll, including his 'serious' mathematical works (under his real name, Charles Dodgson), letters to adults and children, his diaries, formal and whimsical articles in newspapers, we found that the technique, although unable to identify Carroll/Dodgson as the unique author of all the texts, as the authorship attribution literature would demand and expect, seemed to be able to group …

Content truncated to 500 words