Design of an Input Method for Taiwanese Hokkien using Unsupervized Word Segmentation for Language Modeling

Design of an Input Method for Taiwanese Hokkien using Unsupervized Word Segmentation for Language Modeling Pierre Magistry 國立成功大學台灣語文測驗中心 pierre@magistry.fr Abstract This paper presents the challenges and the methodology followed in the design of a new Input Method (IME) for the Taiwanese (Hokkien) language. We first describe the context, the motivations and some of the main issues related to the input of text in Taiwanese on modern computer systems and mobile devices. Then we present the available resources which our system is based on. We will describe the whole architecture of our system. But since the cornerstone of modern IME is the Language Model (LM), the main Natural Language Processing issue on which we will focus in this paper is the estimation of a LM in the case of this under-resourced language. The solution we propose to rely on unsupervised word segmentation which preserves some degree of ambiguity. Keywords: Unsupervized Word Segmentation, Language Modeling, Input Method, Taiwanese 1. Introduction Taiwanese Hokkien (or simply “Taiwanese,” Tâi-gí 台語, throughout the rest of the paper) is a language spoken by a vast majority of the Taiwanese people. It is a Sinitic language of the Minnan (bân-lâm-gí, 閩南語) group. Since our work is based on readily available resources which describe the variety in use in Taiwan, it is better fitted for Taiwanese, but it may be useful to more than 60M people in and outside Taiwan who speak closely related variants. Although this language is still widely spoken in Taiwan, Taiwanese has never been the official language, the efforts in standardization and institutionalization only started in the last decades. The 2016 Conference on Computational Linguistics and Speech Processing ROCLING 2016, pp. 284-298  The Association for Computational Linguistics and Chinese Language Processing 284 Even without state-run institutionalization, written Taiwanese has been in use in printed and handwritten documents for centuries. Depending on the situation, different scripts have been used, including Hàn characters (Hàn-jī), Latin alphabet, adapted versions of Japanese kana and Zhuyin fuhao (注音符號). Nowadays, Hàn-jī and Latin are the two scripts which cover the vast majority of produced texts. Zhuyin is mostly used for annotation of rare Hàn characters or in teaching materials, and for code-mixing in spontaneous writing. Texts written using the Latin script can be divided between different Romanization types, the two more important which are encountered in our resources are Peh-ōe-jī (POJ) and Tâi-lô ̍ (“Taiwan Romanization System”, hereafter TRS). The first one is also called “Church Romanization” due to its origin in missionary works and the latter is recommended by Taiwan’s Ministry of Education since 2006. As a result, the actual situation of written Taiwanese is an interesting case of poly- orthography where one scripter can choose between Hàn-jī and Romanization or (more frequently) mix the two scripts. This requires some specific features from an IME. In the past decades, the status of Taiwanese at school has changed from being forbidden to being taught in classes of “Mother Languages” in primary schools. However the curriculum is still very limited and …

Content truncated to 500 words