Deep Contextualized Self-training for Low Resource Dependency Parsing

Deep Contextualized Self-training for Low Resource Dependency Parsing Guy Rotman and Roi Reichart Faculty of Industrial Engineering and Management, Technion, IIT grotman@campus.technion.ac.il roiri@ie.technion.ac.il Abstract Neural dependency parsing has proven very effective, achieving state-of-the-art results on numerous domains and languages. Unfortunately, it requires large amounts of labeled data, which is costly and laborious to create. In this paper we propose a self- training algorithmthatalleviates this anno- tation bottleneck by training a parser on its own output. Our Deep Contextualized Self-training (DCST) algorithm utilizes representation models trained on sequence labeling tasks that are derived from the parser’s output when applied to unlabeled data, and integrates these models with the base parser through a gating mech- anism. We conduct experiments across multiple languages, both in low resource in-domain and in cross-domainsetups, and demonstrate that DCST substantially out- performs traditionalself-training as wellas recent semi-supervised training methods.1 1 Introduction Deep neural networks (DNNs) have improved the state-of-the-art in a variety of NLP tasks. These include dependency parsing (Dozat and Manning, 2017), semantic parsing (Hershcovich et al., 2017), named entity recognition (Yadav and Bethard, 2018), part of speech (POS) tagging (Plank and Agi´c, 2018), and machine translation (Vaswani et al., 2017), among others. Unfortunately, DNNs rely on in-domain labeled training data, which is costly and laborious to achieve. This annotation bottleneck limits the applicability of NLP technology to a small number of languages and domains. It is hence not a surprise that substantial recent research efforts have been 1Our code is publicly available at https://github. com/rotmanguy/DCST. devoted to DNN training based on both labeled and unlabeled data, which is typically widely available (§ 2). A prominent technique for training machine learning models on labeled and unlabeled data is self-training (Yarowsky, 1995; Abney, 2004). In this technique, after the model is trained on a labeled example set it is applied to another set of unlabeled examples, and the automatically and manually labeled sets are then combined in order to re-train the model—a process that is sometimes performed iteratively. Although self-training has shown useful for a variety of NLP tasks, its success for deep learning models has been quite limited (§ 2). Our goal is to develop a self-training algorithm that can substantially enhance DNN models in cases where labeled training data are scarce. Particularly, we are focusing (§ 5) on the lightly supervised setup where only a small in-domain labeled dataset is available, and on the domain adaptation setup where the labeled dataset may be large but it comes from a different domain than the one to which the model is meant to be applied. Our focus task is dependency parsing, which is essential for many NLP tasks (Levy and Goldberg, 2014; Angeli et al., 2015; Toutanova et al., 2016; Hadiwinoto and Ng, 2017; Marcheggiani et al., 2017), but where self-training has typically failed (§ 2). Moreover, neural dependency parsers (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2017) substantially outperform their linear predecessors, which makes the develop- ment of self-training methods that can enhance these parsers in …

Content truncated to 500 words