Automatic Paragraph Identification: A Study across Languages and Domains

Automatic Paragraph Identification: A Study across Languages and Domains Caroline Sporleder School of Informatics University of Edinburgh 2 Buccleuch Place Edinburgh EH8 9LW, UK csporled@inf.ed.ac.uk Mirella Lapata Department of Computer Science University of Sheffield Regent Court, 211 Portobello Street Sheffield S1 4DP, UK mlap@dcs.shef.ac.uk Abstract In this paper we investigate whether paragraphs can be identifi ed automatically in different languages and domains. We propose a machine learning ap- proach which exploits textual and discourse cues and we assess how well humans perform on this task. Our best models achieve an accuracy that is signifi cantly higher than the best baseline and, for most data sets, comes to within 6% of human per- formance. 1 Introduction Written texts are usually broken up into sentences and paragraphs. Sentence splitting is a necessary pre-processing step for a number of Natural Lan- guage Processing (NLP) tasks including part-of- speech tagging and parsing. Since sentence-fi nal punctuation can be ambiguous (e.g., a period can also be used in an abbreviation as well as to mark the end of a sentence), the task is not trivial and has consequently attracted a lot of attention (e.g., Rey- nar and Ratnaparkhi (1997)). In contrast, there has been virtually no previous research on inferring paragraph boundaries automatically. One reason for this is that paragraph boundaries are usually marked unambiguously by a new line and extra white space. However, a number of applications could bene- fi t from a paragraph detection mechanism. Text- to-text generation applications such as single- and multidocument summarisation as well as text sim- plifi cation usually take naturally occurring texts as input and transform them into new texts satisfying specifi c constraints (e.g., length, style, language). The output texts do not always preserve the struc- ture and editing conventions of the original text. In summarisation, for example, sentences are typ- ically extracted verbatim and concatenated to form a summary. Insertion of paragraph breaks could im- prove the readability of the summaries by indicating topic shifts and providing visual targets to the reader (Stark, 1988). Machine translation is another application for which automatic paragraph detection is relevant. Current systems deal with paragraph boundary in- sertion in the target language simply by preserv- ing the boundaries from the source language. How- ever, there is evidence for cross-linguistic variation in paragraph formation and placement, particularly for languages that are not closely related such as En- glish and Chinese (Zhu, 1999). So, a paragraph in- sertion mechanism that is specifi c to the target lan- guage, instead of one that relies solely on the source language, may yield more readable texts. Paragraph boundary detection is also relevant for speech-to-text applications. The output of auto- matic speech recognition systems is usually raw text without any punctuation or paragraph breaks. This naturally makes the text very hard to read, which can cause processing diffi culties, especially if speech recognition is used to provide deaf stu- dents with real-time transcripts of lectures. Further- more, sometimes the output of a speech recogniser needs to …

Content truncated to 500 words