Classifying English Documents by National Dialect

Classifying English Documents by National Dialect Marco Lui♥♣ and Paul Cook♥ ♥ Department of Computing and Information Systems The University of Melbourne Victoria 3010, Australia ♣ NICTA Victoria Research Laboratory {mhlui,paulcook}@unimelb.edu.au Abstract We investigate national dialect identifica- tion, the task of classifying English doc- uments according to their country of ori- gin. We use corpora of known national origin as a proxy for national dialect. In order to identify general (as opposed to corpus-specific) characteristics of national dialects of English, we make use of a va- riety of corpora of different sources, with inter-corpus variation in length, topic and register. The central intuition is that fea- tures that are predictive of national ori- gin across different data sources are fea- tures that characterize a national dialect. We examine a number of classification ap- proaches motivated by different areas of research, and evaluate the performance of each method across 3 national dialects: Australian, British, and Canadian English. Our results demonstrate that there are lex- ical and syntactic characteristics of each national dialect that are consistent across data sources. 1 Introduction The English language exhibits substantial varia- tion in its usage throughout the world with re- gional differences being noted at the lexical and syntactic levels (e.g., Trudgill and Hannah, 2008) between varieties of English such as that used in Britain and the United States. Although there are many varieties of English throughout the world — including, for example, New Zealand English, African American Vernacular English, and Indian English — there are a smaller number of so-called standard Englishes. British English and Ameri- can English (or North American English) are often taken to be the two main varieties of standard En- glish (Trudgill and Hannah, 2008; Quirk, 1995), with other varieties of standard English, such as Canadian English and Australian English, viewed as more-similar to one of these main varieties. The theme of this work is national dialect iden- tification, the classification of documents as one of a closed set of candidate standard Englishes (here- after referred to as dialects), by exploiting lexical and syntactic variation between dialects. We make use of corpora of text of known national origin as a proxy for text of each dialect. Specifically, we consider Australian English, British English, and Canadian English, three so-called “inner circle” standard Englishes (Jenkins, 2009).1 This preliminary work aims to establish whether standard approaches to text classification are able to accurately predict the variety of stan- dard English in which a document is written. The notion of standard English is differentiated from other factors such as style (e.g., formality) or topic Trudgill (1999), which are expected confounding factors. A model of dialect classification built on a single text type (e.g., standard national corpora) may be classifying documents on the basis of non- dialectal differences such as topic or genre. In order to control for the confounding factors, we utilize text from a variety of sources. By draw- ing training and test data from different sources, the successful transfer of models from one …

Content truncated to 500 words