TnT – A Statistical Part-of-Speech Tagger

TnT -- A Statistical Part-of-Speech Tagger Thorsten Brants Saarland University Computational Linguistics D-66041 Saarbriicken, Germany thorst en@coli, uni-sb, de Abstract Trigrams'n'Tags (TnT) is an efficient statistical part-of-speech tagger. Contrary to claims found elsewhere in the literature, we argue that a tagger based on Markov models performs at least as well as other current approaches, including the Maximum Entropy framework. A recent comparison has even shown that TnT performs significantly better for the tested corpora. We describe the basic model of TnT, the techniques used for smoothing and for handling unknown words. Furthermore, we present evalua- tions on two corpora. 1 Introduction A large number of current language processing sys- tems use a part-of-speech tagger for pre-processing. The tagger assigns a (unique or ambiguous) part-of- speech tag to each token in the input and passes its output to the next processing level, usually a parser. Furthermore, there is a large interest in part-of- speech tagging for corpus annotation projects, who create valuable linguistic resources by a combination of automatic processing and human correction. For both applications, a tagger with the highest possible accuracy is required. The debate about which paradigm solves the part-of-speech tagging problem best is not finished. Recent comparisons of approaches that can be trained on corpora (van Halteren et al., 1998; Volk and Schneider, 1998) have shown that in most cases statistical aproaches (Cut- ting et al., 1992; Schmid, 1995; Ratnaparkhi, 1996) yield better results than finite-state, rule-based, or memory-based taggers (Brill, 1993; Daelemans et al., 1996). They are only surpassed by combinations of different systems, forming a "voting tagger". Among the statistical approaches, the Maximum Entropy framework has a very strong position. Nev- ertheless, a recent independent comparison of 7 tag- gets (Zavrel and Daelemans, 1999) has shown that another approach even works better: Markov mod- els combined with a good smoothing technique and with handling of unknown words. This tagger, TnT, not only yielded the highest accuracy, it also was the fastest both in training and tagging. The tagger comparison was organized as a "black- box test": set the same task to every tagger and compare the outcomes. This paper describes the models and techniques used by TnT together with the implementation. The reader will be surprised how simple the under- lying model is. The result of the tagger comparison seems to support the maxime "the simplest is the best". However, in this paper we clarify a number of details that are omitted in major previous pub- lications concerning tagging with Markov models. As two examples, (Rabiner, 1989) and (Charniak et al., 1993) give good overviews of the techniques and equations used for Markov models and part-of- speech tagging, but they are not very explicit in the details that are needed for their application. We ar- gue that it is not only the choice of the general model that determines the result of the tagger but also the various "small" decisions on alternatives. The aim of this paper is to give a detailed ac- count of …

Content truncated to 500 words