Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7215–7235 July 9-14, 2023 ©2023 Association for Computational Linguistics Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation Benjamin Minixhofer∗1 Jonas Pfeiffer†2 Ivan Vuli´c†3 1Cohere for AI 2Google DeepMind 3University of Cambridge Abstract Many NLP pipelines split text into sentences as one of the crucial preprocessing steps. Prior sentence segmentation tools either rely on punc- tuation or require a considerable amount of sentence-segmented training data: both central assumptions might fail when porting sentence segmenters to diverse languages on a massive scale. In this work, we thus introduce a multi- lingual punctuation-agnostic sentence segmen- tation method, currently covering 85 languages, trained in a self-supervised fashion on unseg- mented text, by making use of newline charac- ters which implicitly perform segmentation into paragraphs. We further propose an approach that adapts our method to the segmentation in a given corpus by using only a small number (64-256) of sentence-segmented examples. The main results indicate that our method outper- forms all the prior best sentence-segmentation tools by an average of 6.1% F1 points. Fur- thermore, we demonstrate that proper sentence segmentation has a point: the use of a (power- ful) sentence segmenter makes a considerable difference for a downstream application such as machine translation (MT). By using our method to match sentence segmentation to the segmen- tation used during training of MT models, we achieve an average improvement of 2.3 BLEU points over the best prior segmentation tool, as well as massive gains over a trivial segmenter that splits text into equally sized blocks. 1 Introduction Sentences are ubiquitous in NLP. Many datasets are made up of annotated sentences (de Marneffe et al., 2021; Aharoni et al., 2019; Conneau et al., 2018, inter alia) and models often expect individual sentences as input (Reimers and Gurevych, 2019, 2020; Liu et al., 2021; Tiedemann and Thottingal, 2020, inter alia). This mandates a need for tools to segment text into sentences: a requirement that ∗Work done during the time BM interned at Cohere. †Equal senior authorship. Collection Sentences UD This is the high season for tourism; | between De- cember and April few people visit and many tour companies and restaurants close down. OPUS100 ’I couldn’t help it,’ said Five, in a sulky tone; ’Seven jogged my elbow.’ | On which Seven looked up and said, ’That’s right, Five! Always lay the blame (...)!’ Ersatz "A lot of people would like to go back to 1970," before program trading, he said. | "I would like to go back to 1970. | But we’re not going back (...)" Table 1: Example sentences of different collections. The pipe (’|’) indicates sentence boundaries. BLEU Segmenter Ground Truth 37.6 Ours 35.3 Ersatz (Wicks and Post, 2021) 26.7 Naïve 22.8 None Figure 1: Impact of sentence segmentation on BLEU scores in MT. Full details in Table 5. typically slips under the radar of many modern NLP systems (Wicks and Post, 2022). Theoretically, a sentence can be …

Content truncated to 500 words