Active Learning for Dependency Parsing with Partial Annotation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 344–354, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Active Learning for Dependency Parsing with Partial Annotation Zhenghua Li†, Min Zhang†∗, Yue Zhang†, Zhanyi Liu‡, Wenliang Chen†, Hua Wu‡, Haifeng Wang‡ † Soochow University, Suzhou, China {zhli13,minzhang,wlchen}@suda.edu.cn, zhangyue1107@qq.com ‡ Baidu Inc., Beijing, China {liuzhanyi,wu hua,wanghaifeng}@baidu.com Abstract Different from traditional active learning based on sentence-wise full annotation (FA), this paper proposes active learning with dependency-wise partial annotation (PA) as a finer-grained unit for dependency parsing. At each iteration, we select a few most uncertain words from an unlabeled data pool, manually annotate their syntactic heads, and add the partial trees into labeled data for parser retraining. Compared with sentence-wise FA, dependency-wise PA gives us more flexibility in task selection and avoids wasting time on annotating trivial tasks in a sentence. Our work makes the following contributions. First, we are the first to apply a probabilistic model to active learning for dependency parsing, which can 1) provide tree probabilities and dependency marginal probabilities as principled uncertainty metrics, and 2) directly learn parameters from PA based on a forest-based training objective. Second, we propose and compare several uncertainty metrics through simulation experiments on both Chinese and English. Finally, we conduct human annotation experiments to compare FA and PA on real annotation time and quality. 1 Introduction During the past decade, supervised dependency parsing has gained extensive progress in boosting parsing performance on canonical texts, especially on texts from domains or genres similar to exist- ing manually labeled treebanks (Koo and Collins, 2010; Zhang and Nivre, 2011). However, the ∗Correspondence author. $0 I1 saw2 Sarah3 with4 a5 telescope6 Figure 1: A partially annotated sentence, where only the heads of “saw” and “with” are decided. upsurge of web data (e.g., tweets, blogs, and product comments) imposes great challenges to existing parsing techniques. Meanwhile, previous research on out-of-domain dependency parsing gains little success (Dredze et al., 2007; Petrov and McDonald, 2012). A more feasible way for open-domain parsing is to manually annotate a certain amount of texts from the target domain or genre. Recently, several small-scale treebanks on web texts have been built for study and evaluation (Foster et al., 2011; Petrov and McDonald, 2012; Kong et al., 2014; Wang et al., 2014). Meanwhile, active learning (AL) aims to reduce annotation effort by choosing and manually an- notating unlabeled instances that are most valu- able for training statistical models (Olsson, 2009). Traditionally, AL utilizes full annotation (FA) for parsing (Tang et al., 2002; Hwa, 2004; Lynn et al., 2012), where a whole syntactic tree is annotated for a given sentence at a time. However, as commented by Mejer and Crammer (2012), the annotation process is complex, slow, and prone to mistakes when FA is required. Particularly, annotators waste a lot of effort on labeling trivial dependencies which can be well handled by cur- rent statistical models (Flannery and Mori, 2015). Recently, researchers report promising results with AL based on partial annotation …

Content truncated to 500 words