Revisiting Tri-training of Dependency Parsers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9457–9473 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9457 Revisiting Tri-training of Dependency Parsers Joachim Wagner and Jennifer Foster School of Computing Dublin City University Dublin, Ireland firstname.lastname@dcu.ie Abstract We compare two orthogonal semi-supervised learning techniques, namely tri-training and pretrained word embeddings, in the task of dependency parsing. We explore language- specific FastText and ELMo embeddings and multilingual BERT embeddings. We focus on a low resource scenario as semi-supervised learning can be expected to have the most impact here. Based on treebank size and available ELMo models, we select Hungar- ian, Uyghur (a zero-shot language for mBERT) and Vietnamese. Furthermore, we include En- glish in a simulated low-resource setting. We find that pretrained word embeddings make more effective use of unlabelled data than tri- training but that the two approaches can be suc- cessfully combined. 1 Introduction Pre-trained neural architectures and contextualised word embeddings are state-of-the-art approaches to combining labelled and unlabelled data in nat- ural language processing tasks taking text as in- put. A large corpus of unlabelled text is processed once and the resulting model is either fine-tuned for a specific task or its hidden states are used as input for a separate model. In the task of depen- dency parsing, recent work is no exception to the above. However, earlier, pre-neural work explored many other ways to use unlabelled data to enrich a parsing model. Among these, self-, co- and tri- training had most impact (Charniak, 1997; Steed- man et al., 2003; McClosky et al., 2006a,b; Søgaard and Rishøj, 2010; Sagae, 2010). Self-training augments the labelled training data with automatically labelled parse trees predicted by a baseline model in an iterative process: 1. Select unlabelled sentences to be parsed in this iteration 2. Parse sentences with current model 3. Optionally discard some of the parse trees, e. g. based on parser confidence 4. Optionally oversample the original labelled data to give it more weight 5. Train a new model on the concatenation of manually labelled and automatically labelled data 6. Check a stopping criterion Co-training proceeds similarly to self-training but uses two different learners, each teaching the other learner, i. e. output of learner A is added to the train- ing data of learner B and vice versa. Tri-training uses three learners and only adds predictions to a learner that the other two learners, the teachers, agree on. As with co-training, the roles of teachers are rotated so that all three learners can receive newly labelled data. We compare tri-training and contextualised word embeddings in the task of dependency parsing, us- ing the same unlabelled data for both approaches. In this comparison, we will try to answer: 1. How does semi-supervised learning with tri- training compare to semi-supervised learning with a combination of context-independent and contextualised word embeddings? 2. Are the above two approaches orthogonal, i. e. do we get an additional boost if we combine them? 3. How do these three approaches …

Content truncated to 500 words