Comparing Methods for Measuring Dialect Similarity in Norwegian

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5343–5350 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5343 Comparing methods for measuring dialect similarity in Norwegian Andre Kåsen*, Janne Bondi Johannessen**, Kristin Hagen**, Anders Nøklestad**, Joel Priestley** * National Library of Norway, ** Department of Linguistics and Scandinavian Studies, University of Oslo * Pb 2674 Solli 0203 Oslo, Norway ** Pb. 1102 Blindern, 0317 Oslo, Norway andre.kasen@nb.no, {j.b.johannessen, kristin.hagen, anders.noklestad, joel.priestley}@iln.uio.no Abstract The present article presents four experiments with two different methods for measuring dialect similarity in Norwegian: the Levenshtein method and the neural long short term memory (LSTM) autoencoder network, a machine learning algorithm. The visual output in the form of dialect maps is then compared with canonical maps found in the dialect literature. All of this enables us to say that one does not need fine-grained transcriptions of speech to replicate classical classification patterns. Keywords: dialectometry, Norwegian, transcription, geolinguistics, maps, the Levenshtein method, neural networks, long short term memory (LSTM) autoencoder network, machine learning algorithm 1. Introduction This paper explores whether it is possible to use existing dialect transcriptions from a recently finished corpus project to automatically generate dialect areas. The phonetic transcriptions are quite coarse-grained, which indicates that they might be possible to generalise over by an automatic method. We have chosen to test two different methods: The Levenshtein method, which uses edit distance to compute distances between dialects, and the neural long short term memory (LSTM) autoencoder network, a machine learning algorithm. The resulting maps show that the transcriptions can indeed be used for the purpose of automatically generating dialect maps, but while the neural network method needs a large dataset, the Levenshtein method can get very good results with a small dataset, too. The paper is structured a follows: Section 2 describes the LIA project from which the transcriptions have been taken, and also describes the actual transcriptions and traditional dialect maps, Section 3 describes the datasets we use, while Sections 4 and 5 present the results of using the two datasets on the two methods. Section 6 concludes the paper, while references are given in Sections 7 and 8. 2. LIA Norwegian - Corpus of Old Dialect Recordings, Transcriptions and Old Maps 2.1 The Project and the Recordings LIA (Language Infrastructure made Accessible) is a national Norwegian project involving the four biggest universities in the country (the University of Oslo – UiO, the University of Bergen – UiB, UiT – The Arctic University of Norway and the Norwegian University for Science and Technology – NTNU), as well as the National Library. It is a five year project 2013–2019 funded by the Research Council of Norway with the aim of digitising dialect recordings at the four universities, and making them into a research infrastructure for research in language, language technology, history and other humanities sciences. The main resources developed in the project are three speech corpora: 1) LIA Norwegian - Corpus of older dialect recordings, …

Content truncated to 500 words