Collecting Tweets to Investigate Regional Variation in Canadian English

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6255–6264 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6255 Collecting Tweets to Investigate Regional Variation in Canadian English Filip Miletic, Anne Przewozny-Desriaux, Ludovic Tanguy CLLE, CNRS & University of Toulouse Toulouse, France {filip.miletic, anne.przewozny, ludovic.tanguy}@univ-tlse2.fr Abstract We present a 78.8-million-tweet, 1.3-billion-word corpus aimed at studying regional variation in Canadian English with a specific focus on the dialect regions of Toronto, Montreal, and Vancouver. Our data collection and filtering pipeline reflects complex design criteria, which aim to allow for both data-intensive modeling methods and user-level variationist sociolinguistic analysis. It specifically consists in identifying Twitter users from the three cities, crawling their entire timelines, filtering the collected data in terms of user location and tweet language, and automatically excluding near-duplicate content. The resulting corpus mirrors national and regional specificities of Canadian English, it provides sufficient aggregate and user-level data, and it maintains a reasonably balanced distribution of content across regions and users. The utility of this dataset is illustrated by two example applications: the detection of regional lexical and topical variation, and the identification of contact-induced semantic shifts using vector space models. In accordance with Twitter’s developer policy, the corpus will be publicly released in the form of tweet IDs. Keywords: Twitter, corpus construction, Canadian English, regional variation 1. Introduction This paper presents a 1.3-billion-word corpus aimed at studying regional variation in Canadian English. It is part of a wider research effort which builds on previous sociolin- guistic studies (Poplack et al., 2006; Boberg, 2012; Rouaud, 2019) and specifically seeks to investigate contact-induced semantic shifts in Quebec English. We aim to identify these linguistic traits by contrasting the English used in Quebec, where the majority of population is French-speaking, to va- rieties of English spoken in Canadian regions where the use of French is limited. Our approach relies on data-intensive methods such as distributional semantic models, but we maintain a variationist sociolinguistic perspective (Labov, 1972; Tagliamonte, 2006) grounded in fine-grained analy- sis of the linguistic behavior of individual speakers. This methodological framework translates to the following corpus design criteria: (1) the corpus should reflect the specificities of the English spoken in Canada, as opposed to corpora of other national varieties of English or more generic datasets; (2) additional geographic metadata is nec- essary to compare different regional varieties of Canadian English: the province of origin of individual texts in the corpus is required as a minimum; (3) each regional subcor- pus must meet a minimum size threshold of ≈ 100 million words in order for the proposed data processing methods to produce reliable results; (4) the reliance of these meth- ods on features such as co-occurrence frequencies entails the need to limit sources of bias such as an irregular dis- tribution of content across authors or a pervasive presence of spam or other types of noise; (5) sociolinguistic analy- sis of ongoing synchronic language variation requires data that is recent, largely contemporaneous, and …

Content truncated to 500 words