A unified approach to sentence segmentation of punctuated text in many languages

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3995–4007 August 1–6, 2021. ©2021 Association for Computational Linguistics 3995 A unified approach to sentence segmentation of punctuated text in many languages Rachel Wicks1 and Matt Post1,2 1Center for Language and Speech Processing 2Human Language Technology Center of Excellence Johns Hopkins University rewicks@jhu.edu, post@cs.jhu.edu Abstract The sentence is a fundamental unit of text pro- cessing. Yet sentences in the wild are com- monly encountered not in isolation, but un- segmented within larger paragraphs and doc- uments. Therefore, the first step in many NLP pipelines is sentence segmentation. Despite its importance, this step is the subject of rel- atively little research. There are no standard test sets or even methods for evaluation, leav- ing researchers and engineers without a clear footing for evaluating and selecting models for the task. Existing tools have relatively small language coverage, and efforts to extend them to other languages are often ad hoc. We introduce a modern context-based mod- eling approach that provides a solution to the problem of segmenting punctuated text in many languages, and show how it can be trained on noisily-annotated data. We also es- tablish a new 23-language multilingual eval- uation set. Our approach exceeds high base- lines set by existing methods on prior English corpora (WSJ and Brown corpora), and also performs well on average on our new evalua- tion set. We release our tool, ERSATZ, as open source. 1 Introduction In many ways, the sentence is the fundamental unit of text in natural language processing (NLP). From the user perspective, tasks such as sentiment analysis, POS tagging, or machine translation con- sume sentences and emit classifications, annota- tions, or transductions of those inputs. Even tasks that operate at the paragraph or document level, such as coreference resolution or summarization, often make use of sentences internally. Yet at the same time, sentences in the wild rarely ex- ist with marked sentence boundaries. For many languages, punctuation serves as a cue for these Examples of Ambiguity in Punctuated Contexts en ... in the U.S. ⊗ House of Representatives ... ...in the U.S. ✓ Most Mexican Spanish ... cs ... podnikanie s.r.o. ⊗ a hlavním investorem ... a Systémy s.r.o. ✓ V roce 2017 ... ro ... W. Pauli s,.a. ⊗ constituie direc¸tii ... ... de Robles s,.a. ✓ A jucat în ... Table 1: Examples of ambiguous FULL STOP punctu- ation in English, Czech, and Romanian from Wikipedia. ✓ denotes a sentence boundary while ⊗ denotes an am- biguous sentence-internal position. boundaries, but this punctuation is ambiguous— as we might see with acronyms or abbreviations in English. When segmented sentences are re- quired, they must be split using a sentence seg- mentation technique that can resolve these ambi- guities. Despite its importance and early position in the NLP pipeline, sentence segmentation is the subject of relatively little research. Widely-used tools such as that in Moses (Koehn et al., 2007) …

Content truncated to 500 words