A Chinese Dataset with Negative Full Forms for General Abbreviation Prediction

A Chinese Dataset with Negative Full Forms for General Abbreviation Prediction Yi Zhang, Xu Sun MOE Key Laboratory of Computational Linguistics, Peking University School of Electronics Engineering and Computer Science, Peking University zhangyi16, xusun@pku.edu.cn Abstract Abbreviation is a common phenomenon across languages, especially in Chinese. In most cases, if an expression can be abbreviated, its abbreviation is used more often than its fully expanded forms, since people tend to convey information in a most concise way. For various language processing tasks, abbreviation is an obstacle to improving the performance, as the textual form of an abbreviation does not express useful information, unless it’s expanded to the full form. Abbreviation prediction means associating the fully expanded forms with their abbreviations. However, due to the deficiency in the abbreviation corpora, such a task is limited in current studies, especially considering general abbreviation prediction should also include those full form expressions that do not have general abbreviations, namely the negative full forms (NFFs). Corpora incorporating negative full forms for general abbreviation prediction are few in number. In order to promote the research in this area, we build a dataset for general Chinese abbreviation prediction, which needs a few preprocessing steps, and evaluate several different models on the built dataset. The dataset is available at https://github.com/lancopku/ Chinese-abbreviation-dataset. Keywords: Chinese abbreviation, negative full forms, conditional random field, long-short term memory 1. Introduction Abbreviation processing mainly consists of three tasks, that is, abbreviation expansion, abbreviation recognition, and abbreviation prediction. Expanding the short form of an expression to its full form is called abbreviation expansion. Extracting the short form and full form pairs from the con- text is called abbreviation recognition. Abbreviation pre- diction refers to predicting the short form of an expression according to its full form. In this paper, we focus on the last task, i.e., abbreviation prediction. Abbreviation predic- tion plays an important role in various language processing tasks, because accurate abbreviation prediction will help improve performance. Sun et al. (2009) shows that better abbreviation prediction will help improve the performance of abbreviation recognition. Abbreviation prediction also benefits other tasks. For example, in an information re- trieval (IR) system, a large number of the web pages con- tain only abbreviations. It will be helpful if we can esti- mate abbreviations of a query, because successful abbrevi- ation prediction may improve the recall of IR systems as Sun et al. (2013a) showed. In addition, Yang et al. (2012) showed that Chinese abbreviation prediction can improve voice-based search quality. 珠穆朗玛峰 珠峰 (Mount Qomolangma) 奥林匹克运动会 奥运会 (Olympic Games) 北京大学 北大 (Peking University ) 清华大学 清华 (Tsinghua University ) 黄金市场 金市 (gold market) Figure 1: Different cases of generating abbreviations English abbreviations are usually formed as acronyms. Studies for English abbreviation proposed various heuris- tics for abbreviation prediction (Park and Byrd, 2001; Wren et al., 2002; Schwartz and Hearst, 2002). For example, use of initials, capital letters, syllable boundaries, stop words, etc. These studies performed well for English abbrevia- tions. While Chinese abbreviations are quite different from English …

Content truncated to 500 words