Dialect Clustering with Character-Based Metrics: in Search of the Boundary of Language and Dialect

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 985–990 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 985 Dialect Clustering with Character-Based Metrics: in search of the boundary of language and dialect Yo Sato, Kevin Heffernan Satoama Language Services, Kwansei Gakuin University Kingston-upon-Thames, U.K., Sanda, Hyogo, Japan satoama@gmail.com, kevin@kwansei.ac.jp Abstract We present in this work a universal, character-based method for representing sentences so that one can thereby calculate the distance between any two sentence pair. With a small alphabet, it can function as a proxy of phonemes, and as one of its main uses, we carry out dialect clustering: cluster a dialect/sub-language mixed corpus into sub-groups and see if they coincide with the conventional boundaries of dialects and sub-languages. By using data with multiple Japanese dialects and multiple Slavic languages, we report how well each group clusters, in a manner to partially respond to the question of what separates languages from dialects. Keywords: clustering, dialects, similar languages, Japanese, Slavic languages, distance metric 1. Introduction ‘A language is a dialect with an army and navy’ is a well- known dictum attributed to sociologist Max Weinrich. It often happens that what are considered to be distinct lan- guages appear more similar to each other than what are considered to be dialects. Our objective, however, despite what the sub-title might suggest, is not a pursuit of a clear distinction. Instead the idea in which this work is oriented is that such distinction should be dissolved into a gradient scale — i.e. that all groupings of languages or dialects are a matter of degree, relative to some metric. In this exploratory work to we experiment with a set of metrics and feature representations on shuffled corpora — classified corpora jumbled into one— for clustering, to see how close the resulting clusters are to the original group- ing. We use two sets of classified corpora, of ‘dialects’ (Japanese dialects) and of similar ‘languages’ (East/South Slavic languages), five-way classified respectively, and re- port how well —in relation to the conventional grouping— these dialects/languages cluster, in a manner in which to partially reply to the question of whether ‘dialects’ can be more different from each other than ‘languages’ are. Clustering of languages/dialects is, in comparison to its su- pervised counterpart, their classification (or identification), a relatively less established field, presumably mainly due to its difficulty to achieve deployment-level performance. Given, however, the reality that digital language resources are often mixed in languages/dialects, the work on their clustering bears not just a theoretical but practical value, as the technique can be used for useful pre-processing to an NLP workflow. Performance aside, another major theme of this work is data representation of a sentence. Clustering standardly re- quires a distance that can be computed on two given dat- apoints and they are represented typically by feature vec- tors, but what would be the best method for the purpose of dialect clustering? If it requires too much human ef- fort …

Content truncated to 500 words