Word-Based Dialect Identification with Georeferenced Rules

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1151–1161, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Word-based dialect identification with georeferenced rules Yves Scherrer LATL Université de Genève Genève, Switzerland yves.scherrer@unige.ch Owen Rambow CCLS Columbia University New York, USA rambow@ccls.columbia.edu Abstract We present a novel approach for (written) di- alect identification based on the discrimina- tive potential of entire words. We generate Swiss German dialect words from a Standard German lexicon with the help of hand-crafted phonetic/graphemic rules that are associated with occurrence maps extracted from a linguis- tic atlas created through extensive empirical fieldwork. In comparison with a character- n-gram approach to dialect identification, our model is more robust to individual spelling dif- ferences, which are frequently encountered in non-standardized dialect writing. Moreover, it covers the whole Swiss German dialect contin- uum, which trained models struggle to achieve due to sparsity of training data. 1 Introduction Dialect identification (dialect ID) can be viewed as an instance of language identification (language ID) where the different languages are very closely re- lated. Written language ID has been a popular re- search object in the last few decades, and relatively simple algorithms have proved to be very successful. The central question of language ID is the following: given a segment of text, which one of a predefined set of languages is this segment written in? Language identification is thus a classification problem. Dialect identification comes in two flavors: spoken dialect ID and written dialect ID. These two tasks are rather different. Spoken dialect ID relies on speech recognition techniques which may not cope well with dialectal diversity. However, the acoustic signal is also available as input. Written dialect ID has to deal with non-standardized spellings that may occult real dialectal differences. Moreover, some phonetic dis- tinctions cannot be expressed in orthographic writing systems and limit the input cues in comparison with spoken dialect ID. This paper deals with written dialect ID, applied to the Swiss German dialect area. An important aspect of our model is its conception of the dialect area as a continuum without clear-cut borders. Our dialect ID model follows a bag-of-words approach based on the assumption that every dialectal word form is defined by a probability with which it may occur in each geographic area. By combining the cues of all words of a sentence, it should be possible to obtain a fairly reliable geographic localization of that sentence. The main challenge is to create a lexicon of dialect word forms and their associated probability maps. We start with a Standard German word list and use a set of phonetic, morphological and lexical rules to obtain the Swiss German forms. These rules are manually extracted from a linguistic atlas. This lin- guistic atlas of Swiss German dialects is the result of decades-long empirical fieldwork. This paper is organized as follows. We start with an overview of relevant research (Section 2) and present the characteristics of the Swiss German di- alect …

Content truncated to 500 words