CorCoDial - Machine translation techniques for corpus-based computational dialectology

CorCoDial – Machine translation techniques for corpus-based computational dialectology Yves Scherrer Olli Kuparinen Department of Digital Humanities, University of Helsinki, Finland firstname.lastname@helsinki.fi Aleksandra Mileti´c Abstract This paper presents CorCoDial, a research project funded by the Academy of Fin- land aiming to leverage machine transla- tion technology for corpus-based compu- tational dialectology. In this paper, we briefly present intermediate results of our project-related research. 1 Introduction Dialectology is concerned with the study of language variation across space. Over the last decades, dialectologists have collected large datasets, which typically consist of transcribed in- terviews with informants. Unfortunately, these interviews cannot easily be compared with each other as they differ considerably in length and con- tent. If informant A does not use word x, this does not necessarily mean that the word does not exist in A’s dialect. It may just be that A chose to talk about topics that did not require the use of word x. The CorCoDial (Corpus-based computational dialectology) project aims to introduce compara- bility in dialect corpora with the help of machine translation techniques. CorCoDial is funded by the Academy of Finland during the period 2021–2025. The core of the project focuses on the dialect- to-standard normalization process, which is a sequence-to-sequence task that maps the phonetic transcriptions to the standardized spellings. We are not only interested in the results of the normaliza- tion process, but also in the emerging representa- tions of dialects and speakers that the (statistical or neural) normalization models learn. These repre- sentations allow us to provide new visualisations of dialect landscapes and to confirm or challenge traditional dialect classifications. Traditional dialect corpora are costly to pro- duce: informants need to be found and inter- © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. viewed, and the recorded interviews need to be transcribed and annotated. To circumvent this data bottleneck, researchers have increasingly turned to user-generated content (UGC), i.e., to texts pub- lished by laypeople on social media. We also investigate to what extent normalization methods trained on “clean” data transcribed by dialectolo- gists generalize to noisier UGC datasets. The main goals of the CorCoDial project are: 1. to improve the automatic normalization of di- alect texts by using state-of-the-art machine translation methods, 2. to extract, visualize, compare and interpret the dialectal patterns emerging from the nor- malization models, and 3. to contrast the patterns found in traditional dialectological corpora with those found in user-generated content. In the following sections, we present some re- sults of our ongoing research. 2 Benchmarking dialect-to-standard normalization systems In contrast to historical text normalization (Boll- mann, 2019; Bawden et al., 2022) and UGC stan- dardization, there have not been any multilin- gual evaluations of dialect-to-standard normaliza- tion systems. In order to establish dialect normal- ization as a distinct task, we compiled a multi- lingual benchmark dataset from existing sources, covering Finnish, Norwegian, Swiss German and Slovene. We evaluate different sequence-to-sequence models that have been previously employed for …

Content truncated to 500 words