Discriminating between Similar Languages on Imbalanced Conversational Texts

Discriminating between Similar Languages on Imbalanced Conversational Texts Junqing He†‡, Xian Huang§, Xuemin Zhao†‡, Yan Zhang†‡, Yonghong Yan†‡♣ †Institute of Acoustics, Chinese Academy of Science, Bejing, China ‡ University of Chinese Acadamy of Sciences, Bejing, China §PLA Information Engineering University Luoyang Division, Henan, China ♣Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences, Urumchi, China hejunqing@hccl.ioa.ac.cn, a77huang852yi@126.com, {zhaoxuemin,zhangyan,yanyonghong}@hccl.ioa.ac.cn Abstract Discriminating between similar languages (DSL) on conversational texts is a challenging task. This paper aims at discriminating between limited-resource languages on short conversational texts, like Uyghur and Kazakh. Considering that Uyghur and Kazakh data are severely imbalanced, we leverage an effective compensation strategy to build a balanced Uyghur and Kazakh corpus. Then we construct a maximum entropy classifier based on morphological features to discriminate between the two languages and investigate the contribution of each feature. Empirical results suggest that our system achieves an accuracy of 95.7% on our Uyghur and Kazakh dataset, which is higher than that of the CNN classifier. We also apply our system to the out-of-domain subtasks of VarDial’2016 DSL shared tasks to test the system’s performance on short conversational texts of other similar languages. Though with much less preprocessing, our system outperforms the champions on both test sets B1 and B2. Keywords: discriminating between similar languages, imbalanced data, morphological features, maximum entropy 1. Introduction Automatic language identification (LID) aims to identi- fy the language a document is written in, which is an important branch in Natural Language Processing (NLP) (Zampieri et al., 2015a). The past two decades had wit- nessed fast development in LID and state-of-the-art sys- tems have achieved high accuracy (Sim˜oes et al., 2014) and wide coverage (Brown, 2014) on standard texts. Howev- er, identifying languages from very little data, from multi- languages input or discriminating between extremely sim- ilar languages are bottlenecks of this field (Ljubeˇsic and Kranjcic, 2014; Zampieri et al., 2015b). What’s more, iden- tifying similar languages with limited resource is unsolved. Uyghur and Kazakh, widely used in Middle East and North West of China, are similar languages. They both belong to the Turkic group of Altaic family and are agglutinative languages. According to Wang et al. (2013), the similari- ty between Uyghur and Kazakh at sentence and word level are over 80% and 90% respectively. They have many char- acteristics in common: (1) They are both written in Arabic alphabets in the right-to-left order. (2) Theoretically there are 32 letters in Uyghur and 33 in Kazakh. The two lan- guages share 26 letters and encoding areas with another 2 letters look exactly the same. (3) There is a large overlap of vocabulary and syntax between the two languages. It is very difficult to identify them by looking up the words in dictionaries. (4) In both languages, a great amount of pre- fixes and suffixes are attached to a word, which makes word stemming and recognition difficult. Here we define ”short conversational texts” as short texts people used to communicate with each other through mo- bile devices, communicational software and …

Content truncated to 500 words