Enhancing Quality of Corpus Annotation: Construction of the Multi-Layer Corpus Annotation and Simplified Validation of the Corpus Annotation

Enhancing Quality of Corpus Annotation: Construction of the Multi-Layer Corpus Annotation and Simplified Validation of the Corpus Annotation Youngbin Noh1, Kuntae Kim1, Minho Lee1, Cheolhun Heo1, Yongbin Jeong1, Yoosung Jeong1, Younggyun Hahm1, Taehwan Oh2,Hyonsu Choe2, Seokwon Park2, Jin-Dong Kim3 and Key-Sun Choi1 1Korea Advanced Institute of Science and Technology, South Korea 2Yonsei University, South Korea, 3Database Center for Life Science, Japan {vincenoh, kuntaek, pathmaker, fairy of 9, kuonom, wjd1004109, hahmyg, kschoi}@kaist.ac.kr {ghksl0604, choehyonsu, sseokhon}@yonsei.ac.kr jdkim@dbcls.rois.ac.jp Abstract In this paper, we construct simultaneously multi-layer corpus annotation of 7 linguistic layers as a gold set. And we design the val- idation procedure using the gold set and re- port the results of a validation procedure for other large-scale corpus annotation of 7 lin- guistic layers. Furthermore, we present a sim- plified validation method without a gold set using annotation models learned by the gold set. Based on the validation results, the ten- dency of annotation across the entire corpus is observed macroscopically, and the corpus an- notation validation results are analyzed micro- scopically to verify the validation methodol- ogy to address the case study. 1 Introduction As a resource for natural language processing, the corpus is annotated with additional information for various purposes. To annotate such various infor- mation to the raw corpus, corpus annotation project must be designed elaborately considering the re- quirements of the annotation procedure, annotation units, annotation tools, human annotators, and so on. A reliable annotation design makes the corpus anno- tation better quality(Finlayson and Erjavec, 2017). Also, the design suitability of the corpus annota- tion project needs to be proved empirically, so the design of the corpus annotation project must be re- vised and supplemented iteratively(Pustejovsky and Stubbs, 2012). In this paper, we construct 7 linguistic layers1(Ide, 1We construct and evaluate the 7 linguistic layer corpus an- 2017) of multi-layered corpus annotation as gold set (210K Eojeol2s) to validate large-scale corpus anno- tation by the 7 layers as evaluation set (3M Eojeols). The gold set is annotated on the subset of the raw corpus of the evaluation set. The annotator groups of gold set by each layer, who annotated gold set assisted by auto-labeling are groups of experts sepa- rated from the annotator groups of evaluation set. We have designed and applied a corpus annotation method that uses the simple inter-annotator agree- ment to construct the gold sets at the same time un- der limited time and human resources. To do this, we assigned one annotation unit to two annotators. According to annotation results from two annota- tors, we conducted the cross-checking process to de- termine the final annotation result. In this paper, validation of evaluation sets by lay- ers is performed by comparing two corpus annota- tions (gold set - evaluation set)3 using the gold set as a criterion. Comparative validation of the two cor- pus annotations using a gold set can only be per- formed on a limited part of the evaluation set shar- ing the same part of raw corpus to be annotated. notation: …

Content truncated to 500 words