Sentence boundary detection: a comparison of paradigms for improving MT quality

Sentence Boundary Detection: A Comparison of Paradigms for Improving MT Quality Daniel J. Walker, David E. Clements, Maki Darwin and Jan W. Amtrup Mendez, Inc. A Division of Lernout & Hauspie, N.V. 5095 Murphy Canyon Road, Suite 300 San Diego, CA 92123, USA {dwalker,dclements,mdarwin,jamtrup}@lhsl.com Abstract The reliable detection of sentence boundaries in running text is one of the first important steps in preparing an input document for translation. Although this is often neglected, it is necessary to obtain a translation with a high degree of quality. In this paper, we present a comparison of different paradigms for the detection of sentence boundaries in written text. We compare three different approaches: Directly encoding the knowledge in a program, a rule-based system relying on regular expressions to describe boundaries, and a statistical maximum-entropy learning algorithm to obtain knowledge about boundaries. Using the statistical system, we obtain a recall of 98.14%, classifying boundaries of six types, and using a training corpus of under 10,000 sentences. Keywords Segmentation, MT for the Web, Maximum-Entropy models, MT Methodologies, Evaluation Introduction The division of running input text into sentences that can be translated in isolation is one of the first important steps for any natural language processing system. In particular, any machine translation system needs reliable information about sentences that present a coherent syntactic portion of text to be analyzed. Consider the following examples of text: (1) Is K.H. Smith here? (2) I bought the apples, pears, lemons, etc. Did you eat them? Two types of problems occur frequently: splitting a single sentence into fragments and falsely conjoining two separate sentences into a single segment. In both cases, our current MT system (L&H Power Translator Pro 7 English-to-French) attempts to parse the ungrammatical input, resulting in poor translation quality. The first sentence translates as (3) Est K.H. Smith ici? The sentence is split into two on the unknown abbreviation, K.H. The translator parses <Is K.H.> as one unit, and <Smith here?> as another. If the input to Power Translator is a similar sentence without an abbreviation, the output is correct French: (4) Est-ce que Kevin Smith est ici? Example (2) above shows the other problem, when two separate sentences are incorrectly conjoined into a single segment. Common abbreviations are kept in a table that the MT engine consults when attempting to disambiguate sentence boundaries. However, because the period is included within the tokenized abbreviation, the parser fails to recognize a sentence boundary when that abbreviation occurs at the end of a sentence. Power Translator joins the two segments and attempts to translate them together: (5) Est-ce que j’ai acheté les pommes, poires, citrons, etc. est-ce que vous les avez mangés? Both sentences are translated as a single question, because of the question mark at the end of the second. If one removes the ambiguous sentence boundary, etc., the MT engine produces a correct parse of both segments: (6) J’ai acheté les pommes, poires, citrons, etc.! Est-ce que vous les avez mangés? These two simple examples shall suffice …

Content truncated to 500 words