Error Measures and Bayes Decision Rules Revisited with Applications to POS Tagging

Error Measures and Bayes Decision Rules Revisited with Applications to POS Tagging Hermann Ney, Maja Popovi´c, David S¨undermann Lehrstuhl f¨ur Informatik VI - Computer Science Department RWTH Aachen University Ahornstrasse 55 52056 Aachen, Germany {popovic,ney}@informatik.rwth-aachen.de Abstract Starting from first principles, we re-visit the statistical approach and study two forms of the Bayes decision rule: the common rule for minimizing the number of string errors and a novel rule for minimizing the number of symbols errors. The Bayes decision rule for minimizing the number of string errors is widely used, e.g. in speech recognition, POS tagging and machine translation, but its justification is rarely questioned. To minimize the number of symbol errors as is more suitable for a task like POS tagging, we show that another form of the Bayes decision rule can be derived. The major purpose of this paper is to show that the form of the Bayes decision rule should not be taken for granted (as it is done in virtually all statistical NLP work), but should be adapted to the error measure being used. We present first experimental results for POS tagging tasks. 1 Introduction Meanwhile, the statistical approach to natural language processing (NLP) tasks like speech recognition, POS tagging and machine translation has found widespread use. There are three ingredients to any statistical approach to NLP, namely the Bayes decision rule, the probability models (like trigram model, HMM, ...) and the training criterion (like maximum likelihood, mutual information, ...). The topic of this paper is to re-consider the form of the Bayes decision rule. In virtually all NLP tasks, the specific form of the Bayes decision rule is never questioned, and the decision rule is adapted from speech recognition. In speech recognition, the typical decision rule is to maximize the sentence probability over all possible sentences. However, this decision rule is optimal for the sentence error rate and not for the word error rate. This difference is rarely studied in the literature. As a specific NLP task, we will consider part- of-speech (POS) tagging. However, the problem addressed comes up in any NLP task which is tackled by the statistical approach and which makes use of a Bayes decision rule. Other prominent examples are speech recognition and machine translation. The advantage of the POS tagging task is that it will be easier to handle from the mathematical point of view and will result in closed- form solutions for the decision rules. From this point-of-view, the POS tagging task serves as a good opportunity to illustrate the key concepts of the statistical approach to NLP. Related Work: For the task of POS tagging, statistical approaches were proposed already in the 60’s and 70’s (Stolz et al., 1965; Bahl and Mercer, 1976), before they started to find widespread use in the 80’s (Beale, 1985; DeRose, 1989; Church, 1989). To the best of our knowledge, the ’standard’ version of the Bayes decision rule, which minimizes the number of string errors, is used in virtually all approaches to POS tagging …

Content truncated to 500 words