A Probabilistic Parser

A PROBABILISTIC PARSER Roger Garside and Fanny Leech Unit for Computer Research on the English Language University of Lancaster Bailrigg Lancaster LAI 4YT, U.K. ABSTRACT The UCREL team at the University of Lancaster is engaged in the development of a robust parsing mechanism, which will assign the appro- priate grammatical structure to sentences in unconstrained English text. The techniques used involve the calculation of probabilities for competing structures, and are based on the techniques successfully used in tagging (i.e. assigning grammatical word classes) to the LOB (Lancaster-Oslo/Bergen) corpus. The first step in the parsing process involves dictionary lookup of successive pairs of gramm- atically tagged words, to give a number of possible continuations to the current parse. Since this lookup will often not be able unambiguously to distinguish the point at which a grammatical constituent should be closed, the second step of the parsing process will have to insert closures and distinguish between alternative parses. It will generate trees representing these possible alternatives, insert closure points for the constituents, and compute a probability for each parse tree from the probability of each constituent within the tree. It will then be able to select a preferred parse or parses for output. The probability of a grammatical constituent is derived from a bank of manually parsed sentences. INTRODUCTION In this paper we present an overview of one part of the work currently being carried out at the Unit for Computer Research on the English Language (UCREL) in the University of Lancaster, under SERC research grant number GR/C/47700. This work involves the automatic syntactic analysis or parsing of the LOB corpus, using the statistical or constituent-likelihood (CL) grammar ideas of Atwell (1983). The work is based on the grammatical tagging of the LOB corpus, both as providing a partially analysed text and because of the techniques used in assigning tags. We therefore begin by briefly describing this earlier project. The grammatical tagging of the LOB corpus is described in detail elsewhere (see, for example, Leech, Garside and Atwell 1983, Marshall 1983, Beale 1985), but in essence there are three stages. The first stage takes the original corpus, on which a certain amount of pre-editing (both automatic and manual) has been performed. It assigns to each word in the corpus a set of possible tags, and it is assumed that the correct tag is in this set. The set of possible tags is chosen without at this stage considering the context in which the word appears, and the choice is made by using an ordered set of decision rules, the most commonly used of which (in about 65-70% of cases) is to look the word up in a dictionary of some 7000 words. The third stage involves looking at those cases where the first stage has resulted in more than one tag being assigned to a word. In this case we calculate the probability of each possible sequence of ambiguous tags, and the most likely sequence is chosen as the correct one. In most …

Content truncated to 500 words