Building Parallel Monolingual Gan Chinese Dialects Corpus

Building Parallel Monolingual Gan Chinese Dialects Corpus Fan XU, Mingwen WANG, Maoxi LI School of Computer Information Engineering, Jiangxi Normal University Nanchang 330022, China {xufan, mwwang, mosesli}@jxnu.edu.cn Abstract Automatic language identification of an input sentence or a text written in similar languages, varieties or dialects is an important task in natural language processing. In this paper, we propose a scheme to represent Gan (Jiangxi province of China) Chinese dialects. In particular, it is a two-level and fine-grained representation using Chinese character, Chinese Pinyin and Chinese audio forms. Guided by the scheme, we manually annotate a Gan Chinese Dialects Corpus (GCDC) including 131.5 hours and 310 documents with 6 different genres, containing news, official document, story, prose, poet, letter and speech, from 19 different Gan regions. In addition, the preliminary evaluation on 2-way, 7-way and 20-way sentence-level Gan Chinese Dialects Identification (GCDI) justifies the appropriateness of the scheme to Gan Chinese dialects analysis and the usefulness of our manually annotated GCDC. Keywords: Parallel Corpus, Monolingual, Gan Chinese Dialects 1. Introduction Automatic language identification of an input sentence or a document is an important task in Natural Language Processing (NLP), especially when processing speech or social media messages. Currently, the interest in language resources and its computational models for the study of similar languages, varieties and dialects has been growing substantially in the last few years (Zampieri et al, 2014, 2015, 2017; Malmasi et al., 2016). Meanwhile, an increasing number of dialect corpus and corresponding computational models have been released for Catalan, Russian, Slovene, etc. However, no free corpus has been released for the similar, varieties or dialects in Mandarin Chinese. Therefore, in this paper, we focus on the corpus building and its computational model design for the closely related parallel monolingual Gan Chinese languages. As we all know, Chinese is spoken in different regions, with distinct differences among regions. There are different expressions for a same concept among the closely related Gan Chinese languages, varieties and dialects. For example, 今里 jin li ‘today’, 今家 jin jia ‘today’, 今宁 jin ning ‘today’, 今兜 jin dou ‘today’, 今朝 jin zhao ‘today’ are the valid expressions in Nanchang, Yichun, Jian, Fuzhou and Yingtan district in Jiangxi province (Gan in short) of China, respectively. Although these expressions are different, they have the same semantic meanings. They all refer to 今天 jin tian ‘today’ in Mandarin Chinese (called 普通话 Putonghua ‘common language’ in Mainland China). More specifically, firstly, we present a scheme to handle Gan Chinese dialects which is a fine-grained representation using Chinese character, Chinese Pinyin and Chinese audio forms. Based on the scheme, we manually annotate a parallel Gan Chinese Dialects Corpus (GCDC) consists of 310 documents with 6 different genres (news, official document, story, prose, poet, letter and speech) from 19 different Gan regions. As a byproduct, the corpus contains the parallel Gan Chinese audio with 131.5 hours. Besides, we conduct a preliminary experiment on the proposed GCDC through the sentence-level Gan Chinese Dialects Identification (GCDI) task. The simple but effective character Chinese …

Content truncated to 500 words