Pause and Stop Labeling for Chinese Sentence Boundary Detection

Proceedings of Recent Advances in Natural Language Processing, pages 146–153, Hissar, Bulgaria, 12-14 September 2011. Pause and Stop Labeling for Chinese Sentence Boundary Detection Hen-Hsen Huang Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan hhhuang@nlg.csie.ntu.edu.tw Hsin-Hsi Chen Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan hhchen@csie.ntu.edu.tw Abstract The fuzziness of Chinese sentence boundary makes discourse analysis more challenging. Moreover, many articles posted on the Internet are even lack of punctuation marks. In this pa- per, we collect documents written by masters as a reference corpus and propose a model to label the punctuation marks for the given text. Conditional random field (CRF) models trained with the corpus determine the correct delimiter (a comma or a full-stop) between each pair of successive clauses. Different tag- ging schemes and various features from differ- ent linguistic levels are explored. The results show that our segmenter achieves an accuracy of 77.48% for plain text, which is close to the human performance 81.18%. For the rich for- matted text, our segmenter achieves an even better accuracy of 82.93%. 1 Introduction To resolve sentence boundary is a fundamental issue for human language understanding. In Eng- lish, sentence boundary detection (SBD) focuses on the disambiguation of the usages of punctua- tion marks such as period to determine if they mark the end of sentences. In Chinese, the concept of “sentences” is fuzz- ier and less-defined. Native Chinese writers sel- dom follow the usage guidelines of punctuation marks. They often decide where to place a pause (i.e., a comma) and where to place a stop (i.e., a full-stop) in the writing according to their indi- vidual subjectivity. People tend to concatenate many clauses with commas. As a result, a Chi- nese sentence is often very long. That makes a text hard to be understood by both humans and machines. For example, a real world sample sen- tence “這是有點霸道，但也有道理，因為他們是 上市公司，每一季要向美國證管會報告總公 司、附屬公司及子公司的營運及財務狀況， 帳都是照一套會計原則來做，所以很多時候 他們的要求，是出自一種單純的需要，而並 不是故意要來欺負我們。” could be divided into three sentences such as “這是有點霸道，但也有道理。” „This is a little overbearing, but is also reasonable.‟ “因為他們是上市公司，每一季要向美國證 管會報告總公司、附屬公司及子公司的營運 及財務狀況，帳都是照一套會計原則來做。” „Because they are listed companies and should report a summary of operation and financial sta- tus of their corporation, subsidiaries, and affili- ates to the U.S. Securities quarterly, the ac- counts are prepared in accordance with the same set of accounting principles.‟ “所以很多時候他們的要求，是出自一種單 純的需要，而並不是故意要來欺負我們。” „For this reason, their requests are usually from the simple need, not to intentionally bully us.‟ The meaning from the set of shorter sentences is more concentrated and more readable than from the single longer one. An even serious issue of Chinese punctuation marking is raised from the massive informal writing on the Internet. The articles posted fre- quently lack of punctuation marks. Authors usu- ally separate clauses by whitespaces and new- line symbols, and the boundaries of sentences are partially or entirely missing. Splitting an entire 146 document into sentences is indispensable. For example, the following text from the Internet “父親在一條小徑裡找到一株相思樹 正結 滿了一小粒一小粒的果實 我終於知道所謂「相思果」是什麼 剪下一兩條樹枝 上面都是纍纍的紅豆 慢慢的從山上走下來 天色也跟著漸漸的黑了” could be divided …

Content truncated to 500 words