Recovering dialect geography from an unaligned comparable corpus

Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH, pages 63–71, Avignon, France, April 23 - 24 2012. c⃝2012 Association for Computational Linguistics Recovering dialect geography from an unaligned comparable corpus Yves Scherrer LATL Universit´e de Gen`eve Geneva, Switzerland yves.scherrer@unige.ch Abstract This paper proposes a simple metric of di- alect distance, based on the ratio between identical word pairs and cognate word pairs occurring in two texts. Different variations of this metric are tested on a corpus contain- ing comparable texts from different Swiss German dialects and evaluated on the basis of spatial autocorrelation measures. The vi- sualization of the results as cluster dendro- grams shows that closely related dialects are reliably clustered together, while mul- tidimensional scaling produces graphs that show high agreement with the geographic localization of the original texts. 1 Introduction In the last few decades, dialectometry has emerged as a field of linguistics that investigates the application of statistical and mathematical methods in dialect research. Also called quanti- tative dialectology, one of its purposes is to dis- cover the regional distribution of dialect similari- ties from aggregated data, such as those collected in dialectological surveys. The work presented here aims to apply dialec- tometric analysis and visualization techniques to a different type of raw data. We argue that classi- cal dialectological survey data are word-aligned by design, whereas our data set, a comparable multidialectal corpus, has to be word-aligned by automatic algorithms. We proceed in two steps. First, we present a cognate identification algorithm that allows us to extract cognate word pairs from the corpus. Then, we measure how many of these cognate word pairs are identical. This ratio gives us a measure of dialectal distance between two texts that is then shown to correlate well with geographic distance. The visualization of the resulting data allows us to recover certain characteristics of the Swiss Ger- man dialect landscape. The paper is structured as follows. In Section 2, the multidialectal corpus is presented. We then discuss how this corpus differs from classical di- alectological data, and how we can use techniques from machine translation to extract the relevant data (Section 3). In Section 4, we define dialect distance as a function of the number of cognate word pairs and identical word pairs. Both types of word pairs are in turn defined by different thresh- olds of normalized Levenshtein distance. Sec- tion 5 deals with the evaluation and visualization of the resulting data, the latter in terms of clus- tering and multi-dimensional scaling. We discuss the results and conclude in Section 6. 2 Data: the Archimob corpus The Archimob corpus used in our experiments is a corpus of transcribed speech, containing texts from multiple Swiss German dialects. The Archimob project was started in 1998 as an oral history project with the aim of gathering and archiving the people’s memory of the Second World War period in Switzerland.1 555 surviving witnesses were interviewed in all Swiss language regions. The interviews of the German-speaking witnesses were conducted in …

Content truncated to 500 words