Improving Sentence Boundary Detection for Spoken Language Transcripts

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7102–7111 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7102 Improving Sentence Boundary Detection for Spoken Language Transcripts Ines Rehbein, Josef Ruppenhofer, Thomas Schmidt Data and Web Science Group Archive for Spoken German University of Mannheim Leibniz Institute for the German Language Mannheim, Germany Mannheim, Germany ines@informatik.uni-mannheim.de, {ruppenhofer|thomas.schmidt}@ids-mannheim.de Abstract This paper presents experiments on sentence boundary detection in transcripts of spoken dialogues. Segmenting spoken language into sentence-like units is a challenging task, due to disfluencies, ungrammatical or fragmented structures and the lack of punctuation. In addition, one of the main bottlenecks for many NLP applications for spoken language is the small size of the training data, as the transcription and annotation of spoken language is by far more time-consuming and labour-intensive than processing written language. We therefore investigate the benefits of data expansion and transfer learning and test different ML architectures for this task. Our results show that data expansion is not straightforward and even data from the same domain does not always improve results. They also highlight the importance of modelling, i.e. of finding the best architecture and data representation for the task at hand. For the detection of boundaries in spoken language transcripts, we achieve a substantial improvement when framing the boundary detection problem as a sentence pair classification task, as compared to a sequence tagging approach. Keywords: Spoken language transcripts, sentence boundary detection, corpus creation 1. Introduction Being able to structure natural spoken discourse into sentence-like units (SLUs) is desirable not only from a the- oretical point of view, but is also a key requirement for enabling research in corpus linguistics as well as the ap- plication of Natural Language Processing tools (e.g. PoS- taggers, syntactic parsers) to transcripts of spoken lan- guage. While various proposals have been made for how to di- vide spoken language in corpora into smaller units, typi- cally these divisions were not guided by syntactic consider- ations. Instead, division into inter-pausal units is common (e.g. Hamaker et al. (1998) for the Switchboard corpus (John J. Godfrey, Edward Holliman, 1993)). For German, the SegCor project presented a proposal and guidelines for dividing transcribed speech into sentence- like units based on Topological Fields (Westpfahl and Gorisch, 2018; Westpfahl et al., 2019). The Topological Fields Model (Drach, 1937; H¨ohle, 1986) is a descriptive grammar formalism that captures regularities in German word order by dividing sentences in different verbal and non-verbal fields and describing their position with regard to the main verb. In a corpus-based study, Schmidt and Westpfahl (2018) then investigated how well the length of gaps between utterances can predict the syntactic bound- aries annotated in the SegCor corpus. They showed that while there is a correlation between gap length and surface syntax, gap length on its own is not sufficient for a reliable prediction of SLU boundaries. Our work builds on previous work on automatic bound- ary detection in German spoken language transcripts (Rup- penhofer …

Content truncated to 500 words