Neural Network for Heterogeneous Annotations

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 731–741, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Neural Network for Heterogeneous Annotations Hongshen Chen∗†§ Yue Zhang ‡ Qun Liu♢† †Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences §University of Chinese Academy of Sciences ‡Singapore University of Technology and Design ♢ ADAPT centre, School of Computing, Dublin City University chenhongshen@ict.ac.cn, yue zhang@sutd.edu.sg Abstract Multiple treebanks annotated under heteroge- neous standards give rise to the research ques- tion of best utilizing multiple resources for im- proving statistical models. Prior research has focused on discrete models, leveraging stack- ing and multi-view learning to address the problem. In this paper, we empirically inves- tigate heterogeneous annotations using neu- ral network models, building a neural network counterpart to discrete stacking and multi- view learning, respectively, finding that neural models have their unique advantages thanks to the freedom from manual feature engineering. Neural model achieves not only better accu- racy improvements, but also an order of mag- nitude faster speed compared to its discrete baseline, adding little time cost compared to a neural model trained on a single treebank. 1 Introduction For many languages, multiple treebanks have been annotated according to different guidelines. For ex- ample, several linguistic theories have been used for defining English dependency treebanks, includ- ing Yamada and Matsumoto (2003), LTH (Johans- son and Nugues, 2007) and Stanford dependencies (De Marneffe et al., 2006). For German, there exist TIGER (Brants et al., 2002) and T¨uBa-D/Z (Telljo- hann et al., 2006). For Chinese, treebanks have been made available under various segmentation granu- larities (Sproat and Emerson, 2003; Emerson, 2005; Xue, 2003). These give rise to the research problem ∗Work done when the first author was visiting SUTD. of effectively making use of multiple treebanks un- der heterogeneous annotations for improving output accuracies (Jiang et al., 2015; Johansson, 2013; Li et al., 2015). The task has been tackled using two typical ap- proaches. The first is based on stacking (Wolpert, 1992; Breiman, 1996; Wu et al., 2003). As shown in Figure 1(a), the main idea is to have a model trained using a source treebank, which is then used to guide a target treebank model by offering source-style fea- tures. This method has been used for leveraging two different treebanks for word segmentation (Jiang et al., 2009; Sun and Wan, 2012) and dependency pars- ing (Nivre and McDonald, 2008; Johansson, 2013). The second approach is based on multi-view learning (Johansson, 2013; Li et al., 2015). The idea is to address both annotation styles simul- taneously by sharing common feature representa- tions. In particular, Johansson (2013) trained depen- dency parsers using the domain adaptation method of Daum´e III (2007), keeping a copy of shared fea- tures and a separate copy of features for each tree- bank. Li et al. (2015) trained POS taggers by cou- pling the labelsets from two different treebanks into a single combined labelset. A summary of such multi-view methods …

Content truncated to 500 words