Generalized Abbreviation Prediction with Negative Full Forms and Its Application on Improving Chinese Web Search

International Joint Conference on Natural Language Processing, pages 641–647, Nagoya, Japan, 14-18 October 2013. Generalized Abbreviation Prediction with Negative Full Forms and Its Application on Improving Chinese Web Search Xu Sun†, Wenjie Li‡, Fanqi Meng‡, Houfeng Wang†, †Key Laboratory of Computational Linguistics (Peking University), Ministry of Education, China ‡Department of Computing, The Hong Kong Polytechnic University xusun@pku.edu.cn cswjli@comp.polyu.edu.hk mengfanqi928@163.com wanghf@pku.edu.cn Abstract In Chinese abbreviation prediction, prior studies are limited on positive full forms. This lab assumption is problematic in real- world applications, which have a large portion of negative full forms (NFFs). We propose solutions to solve this problem of generalized abbreviation prediction. Ex- periments show that the proposed unified method outperforms baselines, with the full-match accuracy of 79.4%. Moreover, we apply generalized abbreviation predic- tion for improving web search quality. Ex- perimental results on web search demon- strate that our method can significantly im- prove the search results, with the search F-score increasing from 35.9% to 64.9%. To our knowledge, this is the first study on generalized abbreviation prediction and its application on web search. 1 Introduction Abbreviations increase the ambiguity in a text. Associating abbreviations with their fully expand- ed forms is important in various natural language processing applications (Pakhomov, 2002; Yu et al., 2006; HaCohen-Kerner et al., 2008). Chi- nese abbreviations represent fully expanded forms (e.g., the left side of Figure 1) through the use of shortened forms (e.g., the right side of Figure 1). Chinese abbreviations are derived via a genera- tive lexical process. Although native speakers may possess intuitions of the generative process, it can- not be adequately explained by any linguistic theo- ry (Chang and Lai, 2004; Chang and Teng, 2007). Abbreviation prediction (i.e., predicting abbre- viation of a given full form) is important in Chi- nese natural language processing applications. For example, it is helpful for information retrieval if we can estimate the abbreviation of a query. For European Economic and Monetary Union 欧洲经济与货币联盟 欧盟 Full Form Abbr. Figure 1: An example of abbreviation prediction. the data of one month’s People’s Daily, only 17% of the documents contain the full form in Figure 1, while more than 70% of the articles contain on- ly the abbreviation in Figure 1. It is expected that abbreviation prediction can improve the re- call in information retrieval. In addition, (Yang et al., 2009b) in speech studies showed that Chinese abbreviation prediction can improve voice-based search quality. The study of Chinese abbreviation prediction is still in an early stage. Chinese abbreviation predic- tion is quite different from English ones, because of its specific characteristics (Sun et al., 2008; Huang et al., 1994; Chang and Teng, 2007; Yang et al., 2009b; Yang et al., 2009a). For example, Chinese abbreviations are not necessarily from the initials of words. They frequently take non-initial characters from the words in the full form. In ad- dition, the Chinese full form does not have word boundaries. To our knowledge, all of the prior studies of Chinese abbreviation prediction (Sun et al., 2008; Sun et …

Content truncated to 500 words