輔助建立混合性系統之自然語言處理系統深度評估平台 — 以評估依存關係分析器為例 (The Platform providing NLP System Deep Comparative Evaluation and Auxiliary Information for Hybrid NLP System Building: Trial on Dependency Parser Evaluation) [In Chinese]

�� � The 2018 Conference on Computational Linguistics and Speech Processing ROCLING 2018, pp. 246-255 ©The Association for Computational Linguistics and Chinese Language Processing The Platform providing NLP System Deep Comparative Evaluation and Auxiliary Information for Hybrid NLP System Building ： Trial on Dependency Parser Evaluation 輔助建立混合性系統之自然語言處理系統深度評估平台 — 以評 估依存關係分析器為例 王億祥 Yi-siang Wang 國立政治大學資訊科學系 Department of Computer Science National Chengchi University 103305079@nccu.edu.tw Abstract This platform includes two main concepts. One, provide multifaceted and deep evaluation for users of Natural Language Processing (NLP) System (such as Syntaxnet or CoreNLP), to help them choose the best one for their needs. Two, independently evaluate single NLP stage of a NLP system (such as pos tagging or dependency parsing), to provide needed auxiliary information for building up a hybrid NLP system (NLP system which composes multiple NLP system for different NLP stages) which is better than a normal NLP system. This paper will be explanation and demonstration centered on these two goals. 摘要 本平台包含兩大概念。 一， 提供自然語言處理(NLP)系統(如Syntaxnet或CoreNLP)的 多面向且深度的衡量，以期協助系統使用者挑選適合其需求的最佳系統。 二，獨立地 評量單一NLP階段(如詞性標注或依存關係分析)，以提供在組建比一般的系統表現更好 的混合型NLP系統(不同的階段使用不同的NLP系統)時，所需的輔助資訊。 本論文將 圍繞此兩個目標展開說明和展示。 Keywords: Evaluation of NLP systems, Evaluation of NLP tools, compare NLP systems, compare NLP tools 關鍵詞： 評估自然語言處理系統，評估自然語言處理工具，自然語言處理系統比較， 自然語言處理工具比較 246 1. Introduction Most NLP projects are based on NLP systems, so the performance of them will deeply affects the result of the projects. But there are so many choices and NLP technology is constantly evolving, which makes normal user hard to get the best NLP system fits their needs. In the previous researches, some didn’t reflect sentence segmentation on their evaluation​[1]​, some evaluate on only one language​[2]​, some evaluate only under special cases​[3]​, and some only focus on LAS and its extension but ignore evaluation on other metrics and factors​[4], [5]​, and some aren’t properly updated to the latest annotation scheme​[6]​. Generally, most of these are lack of a multifaceted and deep evaluation, which means rich choices of evaluation measure, that fit different needs of users or provide evaluations in different perspectives of viewing. Furthermore, we need to be able to evaluate single NLP stage of NLP system, thus we can pick up the best NLP system in the scope of every NLP stage respectively, and concatenate them into hybrid NLP system, which is expected to have better performance than normal NLP system. In this paper, we will first explain our research scope, then briefly introduce evaluation measures for deep and multifaceted evaluation, then comes setting and usage of experiment, finally we will show the evaluation results of the experiment to demonstrate deep evaluation and single NLP stage evaluation for hybrid NLP system building. 2. Scope of this Paper and the Research This paper is about a tool and core concepts behind it, so trivial or detailed things such as implementation of metrics, execution process, will be omitted, because of the page limit. And other thing, like means of evaluating single stage, are omitted because it is largely related to API of the NLP system or it is as the same as other related works do. Additionally, there are many things to discuss …

Content truncated to 500 words