The Present Use of Statistics in the Evaluation of NLP Parsers

| Ii I m m m m m The Present Use of Statistics in the Evaluation of NLP Parsers J.Entwisle D.M.W.Powers Flinders University of South Australia Flinders University of South Australia j im@cs, f linders, edu. au powers~cs, flinders, edu. au Abstract We are concerned that the quality of results produced by an NLP parser bears little, if any, relation to the percentage-results claimed by the various NLP parser-systems presently available for use. To illustrate this problem, we examine one readily available NLP tagging and parsing system, the ENGCG parser; and one tagger, the Brin tagger. We note responses to both artificially generated and naturally oc- curring text. The percentage assessments are methodologically flawed, and should be taken with a grain of salt; instead, assessment of the performance of an NLP parser should be ef- fected by a user, and solely from a considera- tion of the resulting parses of exactly the in- put which an NLP user decides to contribute for such an assessment. Careful attention to input of whatever corpus the user decides on, is presently the only suitable qualifying test of parsing ability. The parsers available are none of them perfectible yet, despite apparent yields now quoted at 99%+. We consider the impact of Zipf's argument of 'least effort' on percent- age assessment; and we open a discussion on estimating the relative complexities of corpora. 1 Introduction Statistics are frequently bandied around in NLP, and would seem to be the obvious way to compare com- peting systems and methodologies. For example: As a rule, data-driven systems rely on statistical generalisations about short se- quences of words or tags....[T]hey tend to reach a 95-97% accuracy [and 12 parsers are referred to.] Interestingly, no signifi- cant improvement beyond the 97% "bar- rier" by means of purely data-driven sys- tems has been reported so far...[Then there is a report of three hybrids - systems that employ linguistic rules for solving some am- biguities - with various additions; and these hybrids] seem capable of exceeding the 97% barrier .... Next, a new system..uses only linguistic distributional rules. Tested against a 38,000-word corpus of previously unseen text, the tagger reaches a better ac- curacy than previous systems (over 99%). (Voutilainen, 1995) We are concerned about the misleading nature of such published statistics, although researchers work- ing on NLP systems are of course well aware that the figures must be interpreted carefully. To be able to show our cause for concern we must look at the statistics of a well known parser, and also at its ac- tions. Unfortunately there are relatively few systems which are freely available for consultation or exami- nation, and for this reason we are forced to pick on some of the few systems that are. The criticisms we make here are not directed against the parsers we use as examples, but against the way in which our field is treating its statistics. Our main example is the ENGCG parser, in one specific form which" has been available for …

Content truncated to 500 words