Challenges and Solutions for Consistent Annotation of Vietnamese Treebank

Challenges and Solutions for Consistent Annotation of Vietnamese Treebank Quy T. Nguyen1&2, Yusuke Miyao1&2, Ha T.T. Le3, Ngan L.T. Nguyen4 1The Graduate University for Advanced Studies (SOKENDAI), Japan 2National Institute of Informatics, Japan 3University of Social Sciences and Humanities, Vietnam 4 University of Information Technology, Vietnam quynt@nii.ac.jp, yusuke@nii.ac.jp, trucha.ussh@gmail.com, ngannlt@uit.edu.vn Abstract Treebanks are important resources for research in natural language processing, speech recognition, theoretical linguistics, etc. To strengthen the automatic processing of the Vietnamese language, a Vietnamese treebank has been built. However, the quality of this treebank is not satisfactory and is a possible source for the low performance of Vietnamese language processing. We have been building a new treebank for Vietnamese with about 40,000 sentences annotated with three layers: word segmentation, part-of-speech tagging, and bracketing. In this paper, we describe several challenges of Vietnamese language and how we solve them in developing annotation guidelines. We also present our methods to improve the quality of the annotation guidelines and ensure annotation accuracy and consistency. Experiment results show that inter-annotator agreement ratios and accuracy are higher than 90% which is satisfactory. Keywords: Vietnamese Treebank, Consistent Annotation, Challenges and Solutions 1. Introduction Treebanks–corpora annotated with syntactic structures, are important resources for researchers in natural language pro- cessing (NLP). Treebanks provide important syntactic in- formation in order to improve the quality of NLP tools. To strengthen the automatic processing of the Vietnamese lan- guage, Nguyen et al. (2009) have built a Vietnamese tree- bank, named VLSP treebank, containing 10,000 sentences. However, the quality of the VLSP treebank, including the quality of the annotation scheme, the annotation guidelines, and the annotation process, is not satisfactory and is a possi- ble source for the low performance of Vietnamese language processing (Nguyen et al., 2012; Nguyen et al., 2013). We have been building a new Vietnamese treebank with 3,000 texts (about 40,000 sentences) covering 14 topics collected from a Vietnamese online newspaper, Thanhnien news1. Our treebank is annotated with three layers: word segmentation (WS), part-of-speech (POS) tagging, and bracketing as showed in Figure 12. We have found that en- suring the annotation consistency and accuracy is one of the most important considerations in the annotation of a tree- bank. This requires clear and complete annotation guide- lines. The guidelines contain the annotation scheme, con- sistent principles to annotate linguistic phenomena, and suf- ficient examples. These documents are not only used to train annotators but also valuable sources serving the uses of the treebank. We prepared three set of guidelines for the Vietnamese tree- bank: WS guidelines, POS tagging guidelines, and bracket- ing guidelines. In this paper, Section 2 describes the general characteristics of the Vietnamese language in comparison 1http://thanhnien.vn 2Underscore "_" is used to link syllables of Vietnamese multi- syllable words. Translation for the Vietnamese word is given as a subscript. If the Vietnamese word does not have a translatable meaning, the subscript is blank. Translation for a Vietnamese sen- tence is given in curly brackets below the original text. Treeing a Vietnamese sentence Original sentence: …

Content truncated to 500 words