Building A Chinese Text Summarizer with Phrasal Chunks and Domain Knowledge

Building a Chinese Text Summarizer with Phrasal Chunks and Domain Knowledge Weiquan Liu & Joe Zhou Intel China Research Center Page 87 ~ 96 Proceedings of Research on Computational Linguistics Conference XIII (ROCLING XIII) Taipei, Taiwan 2000-08-24/2000-08-25 <<RocLing 2000>> Building A Chinese Text Summarizer with Phrasal Chunks and Domain Knowledge Weiquan Liu and Joe Zhou {Lious.Liu; Joe .F.Zhou}@intel.com Intel China Research Center 601 North Tower, Beijing Kerry Center #1 Guanghua Road, Beijing 10002, China Abstract This paper introduces a Chinese summarizier called ThemePicker. Though the system incorporates both statistical and text analysis models, the statistical model plays a major role during the automated process. In addition to word segmentation and proper names identification, phrasal chunk extraction and content density calculation are based on a semantic network pre-constructed for a chosen domain. To improve the readability of the extracted sentences as auto-generated summary, a shallow parsing algorithm is used to eliminate the semantic redundancy. 1 Introduction Due to the overwhelming amount of textual resources over Internet people find it increasingly difficult to grasp targeted information without any adjunctive tools. One of these tools is automatic summarization and abstraction. When coupled with general search and retrieval systems, text summarization can contribute to alleviating the effort in accessing these abundant information resources. It is capable of condensing the amount of original text, enabling the user to quickly capture the main theme of the text. Based on the techniques employed (Hovy, 1998), existing summarization systems can be divided into three categories, i.e., word-frequency-based, cohesion-based, or information-extraction-based. Comparing to the other two techniques the first one is statistical oriented, fast and domain independent (Brandow et al, 1995). The quality, however, is often questionable. Cohesion-based techniques (or sometimes called as being linguistic oriented) can generate more fluent abstracts, but the sentence-by-sentence computation against the entire raw text is often quite expensive. Even the most advanced part of speech (POS) tagging or syntactic parsing algorithms are unable to handle all the language phenomena emerged from giga-bytes of naturally running text. Summarization based on information extraction relies on the predefined templates. It is domain dependent. The unpredictable textual content over Internet, however, may let the templates suffer from incompletion or intra-contradiction no matter how well they might be predefined. In this paper we introduces a Chinese summarization system. Though it is a hybrid system incorporating some natural language techniques, considering the speed and efficiency of text processing we still adapted a statistical oriented algorithm and allowed it to play a major role during the automatic process. After pre-processing, the system first extracts phrasal chunks from the input. The phrasal chunks normally refer to meaningful terms and proper names existing in the text that are difficult to capture using simple methods. Then, we use a domain specific concept network to calculate the content density, i.e. measuring the significance score of each individual sentence. Finally, a Chinese dependency grammar applies as a shallow parser to process the extracted sentences into bracketed frames so as to achieve further binding and embellishment …

Content truncated to 500 words