Bootstrapping POS-taggers using unlabelled data

Bootstrapping POS taggers using Unlabelled Data Stephen Clark, James R. Curran and Miles Osborne School of Informatics University of Edinburgh 2 Buccleuch Place, Edinburgh. EH8 9LW fstephenc,jamesc,osborne g@cogsci.ed.ac.uk Abstract This paper investigates booststrapping part-of- speech taggers using co-training, in which two taggers are iteratively re-trained on each other’s output. Since the output of the taggers is noisy, there is a question of which newly labelled ex- amples to add to the training set. We investi- gate selecting examples by directly maximising tagger agreement on unlabelled data, a method which has been theoretically and empirically motivated in the co-training literature. Our results show that agreement-based co-training can significantly improve tagging performance for small seed datasets. Further results show that this form of co-training considerably out- performs self-training. However, we find that simply re-training on all the newly labelled data can, in some cases, yield comparable results to agreement-based co-training, with only a frac- tion of the computational cost. 1 Introduction Co-training (Blum and Mitchell, 1998), and several vari- ants of co-training, have been applied to a number of NLP problems, including word sense disambiguation (Yarowsky, 1995), named entity recognition (Collins and Singer, 1999), noun phrase bracketing (Pierce and Cardie, 2001) and statistical parsing (Sarkar, 2001; Steedman et al., 2003). In each case, co-training was used successfully to bootstrap a model from only a small amount of labelled data and a much larger pool of un- labelled data. Previous co-training approaches have typ- ically used the score assigned by the model as an indi- cator of the reliability of a newly labelled example. In this paper we take a different approach, based on theoret- ical work by Dasgupta et al. (2002) and Abney (2002), in which newly labelled training examples are selected us- ing a greedy algorithm which explicitly maximises the POS taggers’ agreement on unlabelled data. We investigate whether co-training based upon di- rectly maximising agreement can be successfully ap- plied to a pair of part-of-speech (POS) taggers: the Markov model TNT tagger (Brants, 2000) and the max- imum entropy C&C tagger (Curran and Clark, 2003). There has been some previous work on boostrap- ping POS taggers (e.g., Zavrel and Daelemans (2000) and Cucerzan and Yarowsky (2002)), but to our knowledge no previous work on co-training POS taggers. The idea behind co-training the POS taggers is very simple: use output from the TNT tagger as additional labelled data for the maximum entropy tagger, and vice versa, in the hope that one tagger can learn useful infor- mation from the output of the other. Since the output of both taggers is noisy, there is a question of which newly labelled examples to add to the training set. The addi- tional data should be accurate, but also useful, providing the tagger with new information. Our work differs from the Blum and Mitchell (1998) formulation of co-training by using two different learning algorithms rather than two independent feature sets (Goldman and Zhou, 2000). Our results show that, when using very small amounts of manually …

Content truncated to 500 words