Learning and Application of Differential Grammars

Learning and Application of Differential Grammars David M. W. Powers Artificial Intelligence Laboratory Department of Computer Science The Flinders University of South Australia powers~acm, org Abstract We examine the Differential Grammar, a representation designed to discriminate which of a set of eonfusable alternatives is most likely in the context it occurs in. This approach is useful whereever uncertainty may exist about the identity of a token or sequence of tokens, including in speech recognition, optical character recognition and machine translation. In this paper our application is word processing: we discuss multiple models of confusion which may be used in the identification of confused words, we show how significant contexts may be identified and condensed into Dif- ferential Grammars, and we contrast the performance of our implementation with that of two commercial grammar checkers which purport to handle the confused word problem. 1 Introduction In this paper, we explore the concept of a Differential Grammar and apply it to the problem of grammar checking - in the sense that it is used on the box your WYSIWYG word processor came in! A Differential Grammar is not a grammar in the traditional rule- oriented sense, and although it is lexically-focussed it doesn't really have a concept of a rule at all, but is somewhat more specialized, and extremely sim- ple. We introduce a simple statistical preclassifica- tion, which we use to define a modified Ngram envi- ronment for each member of a confusion class, and then adjust the size of our environment until a pre- determined significance level or diameter is reached. We also demonstrate several approaches to the au- tomated generation of the confusion classes. The structure of the paper is as follows: we in- troduce Differential Grammars and motivate them in terms of grammar checking, we discuss the acqui- sition, efficiency and significance issues, we present an improved user interface for grammar checkers, and we demonstrate the improvement achieved by Differential Grammar based checkers compared with commercial products. Finally, we discuss other ex- periments relating to automated acquisition of dif- ferential grammars. 2 Commonly Confused Words A word processor's grammar checker typically checks grammar in two senses, a pre-/proscriptive sense which should be characterized by what are more properly identified as style rules, and a spell- checking like sense which is characterized as the use of grammar rules to identify typos, substitutions, omissions, duplications which occur at the word or letter level. The errors we focus on are those 'typos' which re- sult in one word being substituted by another word " - both of which are words which would be accepted by the spell-checker. This is closely related to the problems of commonly confused words, homophones and near homophones, but the most important dif- ference between the different types of substitution is in fact whether readers, proofreading their own work, or looking at a detected error, will recognize that it is an error or not. This is not a new problem, either from the point of view of commercial …

Content truncated to 500 words