Creating dialect sub-corpora by clustering: a case in Japanese for an adaptive method

Creating Dialect Sub-corpora by Clustering: a case in Japanese for an adaptive method Yo Sato, Kevin Heffernan Satoama Language Services, Kwansei Gakuin University Kingston-upon-Thames, U.K., Sanda, Hyogo, Japan yo@satoama.co.uk, kevin@kwansei.ac.uk Abstract We propose a pipeline through which to derive clusters of dialects, given a mixed corpus composed of different dialects, when their standard counterpart is sufficiently resourced. The test case is Japanese, where the written standard language is sufficiently equipped with adequate resources. Our method starts by detecting non-standard contents first, and then clusters what is deemed dialectal. We report the results on the clustering of mixed Twitter corpus into four dialects (Kansai, Tohoku, Chugoku and Kyushu). Keywords: dialect, Japanese, clustering, adaptation of language model 1. Introduction In what follows we propose a pipeline through which to derive clusters for dialects, given a body of ‘mixed’ cor- pus composed of different dialects, when one of them is sufficiently resourced. The main component of the proposal is an unsupervised clustering method to gen- erate dialectal sub-corpora. The test case we experi- ment on is Japanese, where the variety deemed ‘stan- dard’, the Tokyo dialect, is equipped with an adequate language model. We call such a dominant and well- resourced language a ‘pivot’ language, and its model a pivot model. Our main target data are Twitter utter- ances, coming from four broad dialect-speaking regions of Japan outside the Tokyo area. We will show how a) the dialectal content can be identified in this dataset and b) the identified dialectal content can be classified into the four dialects. While our experiments are specifically on Japanese, we believe the work will have a wider implication, since a similar situation exists in many linguistic communities, where a well-resourced pivot exists but dialects lack mature enough language models. Ironically enough this is despite the fact that the availability of dialect data has increased in the cyberspace, with the rise of social media and interactive message boards. It is prin- cipally the lack of classified data that prevents the data from being utilised. Japanese poses an additional challenge since there is no word segmentation in its orthography. This challenge however is a general problem. Word segmentation is an artefact of orthography, which is not present in speech, and hence requires lexical knowledge to perform. Thus the challenge is similar to the situation faced by a di- alect monolingual who is exposed to other dialects in a spoken form: the difficulty is not so much with the un- known ‘words’ as the incomprehensible ‘chunks’ that may or may not correspond to words. Our proposed procedure consists of three main stages. At the first stage we set apart the portion that is de- viant in the view of the word pivot model. We find that this separated part not only contains dialects but non-dialectal non-standard utterances (e.g. internet jargon), so a second filtering follows, to set apart gen- uinely dialectal sentences. Here, in contrast to the prior stage, the character pivot model is used for fil- tering. The final …

Content truncated to 500 words