DIMSIM: An Accurate Chinese Phonetic Similarity Algorithm Based on Learned High Dimensional Encoding

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 444–453 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 444 DIMSIM: An Accurate Chinese Phonetic Similarity Algorithm based on Learned High Dimensional Encoding Min Li IBM Research minli@us.ibm.com Marina Danilevsky IBM Research mdanile@us.ibm.com Sara Noeman IBM noemans@eg.ibm.com Yunyao Li IBM Research yunyaoli@us.ibm.com Abstract Phonetic similarity algorithms identify words and phrases with similar pronunciation which are used in many natural language process- ing tasks. However, existing approaches are designed mainly for Indo-European languages and fail to capture the unique properties of Chinese pronunciation. In this paper, we pro- pose a high dimensional encoded phonetic similarity algorithm for Chinese, DIMSIM. The encodings are learned from annotated data to separately map initial and final phonemes into n-dimensional coordinates. Pinyin pho- netic similarities are then calculated by aggre- gating the similarities of initial, final and tone. DIMSIM demonstrates a 7.5X improvement on mean reciprocal rank over the state-of-the- art phonetic similarity approaches. 1 Introduction Performing the mental gymnastics of transform- ing ‘I’m hear’ to ‘I’m here,’ or, ‘I can’t so but- tons’ to ‘I can’t sew buttons,’ is familiar to anyone who has encountered autocorrected text messages, punny social media posts, or just friends with bad grammar. Although at first glance it may seem that phonetic similarity can only be quantified for audible words, this problem is often present in purely textual spaces, such as social media posts or text messages. Incorrect homophones and syn- ophones, whether used in error or in jest, pose challenges for a wide range of NLP tasks, such as named entity identification, text normalization and spelling correction (Chung et al., 2011; Xia et al., 2006; Toutanova and Moore, 2002; Twiefel et al., 2014; Lee et al., 2013; Kessler, 2005). These tasks must therefore successfully transform incor- rect words or phrases (‘hear’,’so’) to their phonet- ically similar correct counterparts (’here’,’sew’), which in turn requires a robust representation of phonetic similarity between word pairs. A reli- Pinyin initial final tone xi1 x i 1 fan4 f an 4 Table 1: Example Pinyins. 偶(ou2,我wo2)稀饭(xi1fan4,喜欢xi2huan1)你。 I like you. 杯具(bei1ju4,悲剧bei1ju4)啊，为一个女孩纸 (zhi2,子zi5)这么香菇(xiang1gu1,想哭 xiang2ku1)。 Sadly, I am heart broken for a girl. Table 2: Microblogs using phonetic transcription. able approach for generating phonetically simi- lar words is equally crucial for Chinese text (Xia et al., 2006). Unfortunately, most existing phonetic similar- ity algorithms such as Soundex (Archives and Administration, 2007) and Double Metaphone (DM) Philips (2000) are motivated by English and designed for Indo-European languages. Words are encoded to approximate phonetic presentations by ignoring vowels (except foremost ones), which is appropriate where phonetic transcription consists of a sequence of phonemes, such as for English. In contrast, the speech sound of a Chinese char- acter is represented by a single syllable in Pinyin consisting of two or three parts: an initial (op- tional), a final or compound finals, and tone 1 (Ta- ble 1). As a result, phonetic similarity approaches designed for Indo-European languages often fall short when applied …

Content truncated to 500 words