Optical Character Recognition, Word Segmentation, Sentence Segmentation, and Information Extraction for Historical and Literature Texts in Classical Chinese

Mitigating Impacts of Word Segmentation Errors on Collocation Extraction in Chinese 廖永賦 Yongfu Liao 國立臺灣大學語言學研究所 Graduate Institute of Linguistics National Taiwan University liao961120@gmail.com 謝舒凱 Shu-Kai Hsieh 國立臺灣大學語言學研究所 Graduate Institute of Linguistics National Taiwan University shukaihsieh@ntu.edu.tw 摘要 隨著網路的盛行，自動斷詞與標記的大規模語料庫逐漸普及。自動化不可避免地引入 一些斷詞與標記的錯誤，並可能對下游任務產生負面影響。搭配詞的自動抽取是一項 受斷詞品質影響的任務。本文探討一些方法試圖減輕斷詞錯誤對漢語搭配詞抽取之影 響。我們嘗試了一個結合多個共現訊息的簡單線性模型，試圖減少抽取出之搭配詞含 有的斷詞錯誤。實驗結果顯示，此模型無法區分搭配詞是否為斷詞錯誤所導致。因此 ，我們使用了 FastText 詞向量的訊息進行了另一個案例研究。結果顯示，由斷詞錯誤 所產生的假搭配詞與真正的搭配詞，其之間的語義相似性具有不同的特徵。未來研究 可嘗試在搭配詞抽取中加入詞向量的訊息。 Abstract The prevalence of the web has brought about the construction of many large-scale, automatically segmented and tagged corpora, which inevitably introduces errors due to automation and are likely to have negative impacts on downstream tasks. Collocation extraction from Chinese corpora is one such task that is profoundly influenced by the quality of word segmentation. This paper explores methods to mitigate the negative impacts of word segmentation errors on collocation extraction in Chinese. In particular, we experimented with a simple model that aims to combine several association measures linearly to avoid retrieving false collocations resulting from word segmentation errors. The results of the experiment show that this simple model could not differentiate between true collocations and false collocations The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24–26, 2020. The Association for Computational Linguistics and Chinese Language Processing 8 resulting from word segmentation errors. An ad hoc case study incorporating information from FastText word vectors is also conducted. The results show that collocates resulting from correct and erroneous word segmentation have different profiles in terms of the semantic similarities between the collocates. The incorporation of word vector information to differentiate between true and false collocations is suggested for future work. 關鍵詞：搭配詞抽取、中文斷詞、詞向量 Keywords: Collocation Extraction, Chinese Word Segmentation, Word Vector 1 Introduction A collocation, in Firthian sense, is a combination of words that tend to occur near each other in natural language [1]. To measure the tendency for words to co-occur, various statistical measures are proposed to quantify the association strengths of word pairs. These association measures are often used to rank and extract collocations from corpora. As the concept of collocation was developed in the western world, which has a writing system that clearly delimits word boundaries, the adoption of the concept of collocation in languages where the notion of wordhood is not clear necessitates a preprocessing step that segments the text into sequences of “words”. Computing association measures based on the segmented text to extract collocations thus requires an additional assumption—the word segmentation must return correct results. Otherwise, the collocations extracted might be nonsense—instead of being recurrent “word” combinations, the “collocations” may in fact be “character” combinations that have a tendency to co-occur. With the prevalence of the internet, large corpora constructed from texts collected from the web has become common. At the same time, manual checking of the automatic segmentation and tagging of the corpora to ensure the quality has become nearly impossible, as the amount of data collected is enormous. In addition, out-of-vocabulary words such as named entities, new terms, and special usage of particular subcultures frequently appear in web texts [2], further casting doubt on the performance of automatic segmentation of the constructed corpora. Since …

Content truncated to 500 words