Lattice-Based Word Identification in CLARE

LATTICE-BASED WORD IDENTIFICATION IN CLARE David M. Carter SRI International Cambridge Computer Science Research Centre 23 Millers Yard Cambridge CB2 1RQ, U.K. dmc@cam, sri. com ABSTRACT I argue that because of spelling and typing errors and other properties of typed text, the identification of words and word boundaries in general requires syntactic and semantic knowledge. A lattice representation is there- fore appropriate for lexical analysis. I show how the use of such a representation in the CLARE system allows different kinds of hy- pothesis about word identity to be integrated in a uniform framework. I then describe a quantitative evaluation of CLARE's perfor- mance on a set of sentences into which ty- pographic errors have been introduced. The results show that syntax and semantics can be applied as powerful sources of constraint on the possible corrections for misspelled words. 1 INTRODUCTION In many language processing systems, uncer- tainty in the boundaries of linguistic units, at various levels, means that data are repre- sented not as a well-defined sequence of units but as a lattice of possibilities. It is common for speech recognizers to maintain a lattice of overlapping word hypotheses from which one or more plausible complete paths are subse- quently selected. Syntactic parsing, of either spoken or written language, frequently makes use of a chart or well-formed substring ta- ble because the correct bracketing of a sen- tence cannot (easily) be calculated determin- istically. And lattices are also often used in the task of converting Japanese text typed in kana (syllabic symbols) to kanji; the lack of in- terword spacing in written Japanese and the complex morphology of the language mean that lexical items and their boundaries cannot be reliably identified without applying syntac- tic and semantic knowledge (Abe et al, 1986). In contrast, however, it is often assumed that, for languages written with interword spaces, it is sufficient to group an input char- acter stream deterministically into a sequence of words, punctuation symbols and perhaps other items, and to hand this sequence to the parser, possibly after word-by-word mor- phological analysis. Such an approach is sometimes adopted even when typographi- cally complex inputs are handled; see, for ex- ample, Futrelle et al, 1991. In this paper I observe that, for typed in- put, spaces do not necessarily correspond to boundaries between lexical items, both for lin- guistic reasons and because of the possibil- ity of typographic errors. This means that a lattice representation, not a simple sequence, should be used throughout front end (pre- parsing) analysis. The CLARE system under development at SRI Cambridge uses such a representation, allowing it to deal straightfor- wardly with combinations or multiple occur- rences of phenomena that would be difficult or impossible to process correctly under a se- quence representation. As evidence for the performance of the approach taken, I describe 159 an evaluation of CLARE's ability to deal with typing and spelling errors. Such errors are es- pecially common in interactive use, for which CLARE is designed, and the correction of …

Content truncated to 500 words