Mining and Modeling Relations between Formal and Informal Chinese Phrases from Web Corpora

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1031–1040, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Mining and Modeling Relations between Formal and Informal Chinese Phrases from Web Corpora Zhifei Li and David Yarowsky Department of Computer Science and Center for Language and Speech Processing Johns Hopkins University, Baltimore, MD 21218, USA zhifei.work@gmail.com and yarowsky@cs.jhu.edu Abstract We present a novel method for discovering and modeling the relationship between in- formal Chinese expressions (including collo- quialisms and instant-messaging slang) and their formal equivalents. Specifically, we pro- posed a bootstrapping procedure to identify a list of candidate informal phrases in web corpora. Given an informal phrase, we re- trieve contextual instances from the web us- ing a search engine, generate hypotheses of formal equivalents via this data, and rank the hypotheses using a conditional log-linear model. In the log-linear model, we incorpo- rate as feature functions both rule-based intu- itions and data co-occurrence phenomena (ei- ther as an explicit or indirect definition, or through formal/informal usages occurring in free variation in a discourse). We test our system on manually collected test examples, and find that the (formal-informal) relation- ship discovery and extraction process using our method achieves an average 1-best preci- sion of 62%. Given the ubiquity of informal conversational style on the internet, this work has clear applications for text normalization in text-processing systems including machine translation aspiring to broad coverage. 1 Introduction Informal text (e.g., newsgroups, online chat, blogs, etc.) is the majority of all text appearing on the Inter- net. Informal text tends to have very different style from formal text (e.g., newswire, magazine, etc.). In particular, they are different in vocabulary, syn- tactic structure, semantic interpretation, discourse Formal Informal �� (BaiBai)[bye-bye] 88 (BaBa) �� (XiHuan)[like] �� (XiFan)[gruel] �� (GeGe)[elder brother] GG �� (GeMi)[fans] �� (FenSi)[a food] Table 1: Example Chinese Formal-informal Relations. The PinYin pronunciation is in parentheses and an op- tional literal gloss is in brackets. structure, and so on. On the other hand, certain re- lations exist between the informal and formal text, and informal text often has a viable formal equiva- lent. Table 1 shows several naturally occurring ex- amples of informal expressions in Chinese, and Ta- ble 2 provides a more detailed inventory and charac- terization of this phenomena1. The first example of informal phrase “88” is used very often in Chinese on-line chat when a person wants to say “bye-bye” to the other person. This can be explained as fol- lows. In Chinese, the standard equivalent to “bye- bye” is “��” whose PinYin is “BaiBai”. Coin- cidentally, the PinYin of “88” is “BaBa”. Because “BaBa” and “BaiBai” are near homophones, people often use “88” to represent “��”, either for input convenience or just for fun. The other relations in Table 1 are formed due to similar processes as will be described later. Due to the often substantial divergence between 1For clarity, we represent Chinese words in the format: Chi- nese characters (optional PinYin equivalent in parentheses and …

Content truncated to 500 words