Handling Noisy Training and Testing Data

Handling noisy training and testing data Don Blaheta Department of Computer Science Brown University dpb@cs.brown.edu Abstract In the field of empirical natural language processing, researchers constantly deal with large amounts of marked-up data; whether the markup is done by the researcher or someone else, human nature dictates that it will have errors in it. This paper will more fully characterise the problem and discuss whether and when (and how) to correct the errors. The discussion is illustrated with specific examples involving function tagging in the Penn treebank. 1 Introduction: Errors Nobody’s perfect. A clich´e, but in the field of empir- ical natural language processing, we know it to be true: on a daily basis, we work with large corpora created by, and often marked up by, humans. Falli- ble as ever, these humans have made errors. For the errors in content, be they spelling, syntax, or some- thing else, we can hope to build more robust systems that will be able to handle them. But what of the errors in markup? In this paper, we propose a system for cataloguing corpus errors, and discuss some strategies for dealing with them as a research community. Finally, we will present an example (function tagging) that demon- strates the appropriateness of our methods. 2 An error taxonomy 2.1 Type A: Detectable errors The easiest errors, which we have dubbed “Type A”, are those that can be automatically detected and fixed. These typically come up when there would be multiple reasonable ways of tagging a certain in- teresting situation: the markup guidelines arbitrarily choose one, and the human annotator unthinkingly uses the other. hhh �� XX XX  9 the dealers by NP ...should go here This LGS tag... VBN PP-LGS IN started VP Figure 1: A function tag error of Type A The canonical example of this sort of thing is the treebank’s LGS tag, representing the “logical sub- ject” of a passive construction. It makes a great deal of sense to put this tag on the NP object of the ‘by’ construction; it makes almost as much sense to tag the PP itself, especially since (given a choice) most other function tags are put there. The tree- bank guidelines specifically choose the former: “It attaches to the NP object of by and not to the PP node itself.” (Bies et al., 1995) Nevertheless, in sev- eral cases the annotators put the tag on the PP, as shown in Figure 1. We can automatically correct this error by algorithmically removing the LGS tag from any such PP and adding it to the object thereof. The unifying feature of all Type A errors is that the annotator’s intent is still clear. In the LGS case, the annotator managed to clearly indicate the pres- ence of a passive construction and its logical subject. Since the transformation from what was marked to what ought to have been marked is straightforward and algorithmic, we can easily apply this correction to all data. 2.2 Type B: Fixable errors …

Content truncated to 500 words