TweetGeo - A Tool for Collecting, Processing and Analysing Geo-encoded Linguistic Data

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3412–3421, Osaka, Japan, December 11-17 2016. TweetGeo – A Tool for Collecting, Processing and Analysing Geo-encoded Linguistic Data Nikola Ljubeˇsi´c Dept. of Knowledge Technologies Joˇzef Stefan Institute nikola.ljubesic@ijs.si Tanja Samardˇzi´c CorpusLab, URPP Language and Space University of Zurich tanja.samardzic@uzh.ch Curdin Derungs GISLab, URPP Language and Space University of Zurich curdin.derungs@geo.uzh.ch Abstract In this paper we present a newly developed tool that enables researchers interested in spatial variation of language to define a geographic perimeter of interest, collect data from the Twitter streaming API published in that perimeter, filter the obtained data by language and country, define and extract variables of interest and analyse the extracted variables by one spatial statistic and two spatial visualisations. We showcase the tool on the area and a selection of languages spoken in former Yugoslavia. By defining the perimeter, languages and a series of linguistic variables of interest we demonstrate the data collection, processing and analysis capabilities of the tool. 1 Introduction Geographic distribution of linguistic features is traditionally studied in dialectology (regarding closely- related varieties) and in language typology (regarding different languages and language families), with the goal of identifying the patterns of language change. The potential for studying the geographic spread of linguistic features increased with the development of computer-mediated-communication (CMC). Short and long texts produced by the users of social media communication platforms constitute large samples of authentic language use that can be automatically retrieved and analysed to address a range of questions about human behavior, including spatial linguistic patterns analysed in this paper. The social network Twitter made an especially important contribution to the development of new methods of collecting linguistic data from the Internet by allowing access to the content produced by their users through an API (application programming interface). One interesting feature of Twitter is that tweets are often associated with spatial information, explicitly (GPS coordinates) or implicitly (place names). The data collected from Twitter can either be used as a valuable complement to linguistic data already collected by traditional means or, if traditional data is not available, as a replacement. This oppor- tunity, however, comes with considerable challenges. First, using the data collection interface requires technical skills that researchers interested in studying language variation usually cannot be expected to have. Second, once collected, the data often turns out to be noisy, difficult to annotate and unevenly distributed in space. Observing patterns therefore requires advanced processing methods, such as spatial statistics and geographic information science (GIScience). In this paper, we present a tool set that combines computational linguistics and GIScience methods in order to facilitate the collection, visualisation and analysis of georeferenced tweets. Our major goal is to allow the wider linguistic research community access to data automatically collected from the Internet (Twitter data in this particular case). We provide a configurable tool set for speeding up the research process without limiting researchers in their choice of input data and analysis techniques. The user is, …

Content truncated to 500 words