Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1813–1822, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1813 Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection Shaolei Wang, Zhongyuan Wang, Wanxiang Che∗, Ting Liu Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China {slwang, zywang, car, tliu}@ir.hit.edu.cn Abstract Most existing approaches to disfluency detec- tion heavily rely on human-annotated corpora, which is expensive to obtain in practice. There have been several proposals to alleviate this is- sue with, for instance, self-supervised learn- ing techniques, but they still require human- annotated corpora. In this work, we explore the unsupervised learning paradigm which can potentially work with unlabeled text corpora that are cheaper and easier to obtain. Our model builds upon the recent work on Noisy Student Training, a semi-supervised learning approach that extends the idea of self-training. Experimental results on the commonly used English Switchboard test set show that our approach achieves competitive performance compared to the previous state-of-the-art su- pervised systems using contextualized word embeddings (e.g. BERT and ELECTRA). 1 Introduction Automatic speech recognition (ASR) outputs often contain various disfluencies, which is a character- istic of spontaneous speech and create barriers to subsequent text processing tasks like parsing, ma- chine translation, and summarization. Disfluency detection (Zayats et al., 2016; Wang et al., 2016; Wu et al., 2015) focuses on recognizing the disflu- encies from ASR outputs. As shown in Figure 1, a standard annotation of the disfluency structure indicates the reparandum (words that the speaker intends to discard), the interruption point (denoted as ‘+’, marking the end of the reparandum), an optional interregnum (filled pauses, discourse cue words, etc.) and the associated repair (Shriberg, 1994). Ignoring the interregnum, disfluencies are cate- gorized into three types: restarts, repetitions and *Email corresponding. a flight [ to Boston + {um I mean} to Denver ] RM IM RP Figure 1: A sentence from the English Switchboard corpus with disfluencies annotated. RM=Reparandum, IM=Interregnum, RP=Repair. The preceding RM is corrected by the following RP. Type Annotation repair [ the + they ’re ] voice activated repair [ we want + {well} in our area we want ] to repetition [we got + {uh} we got ] to talking restart [ we would like + ] let’s go to the Table 1: Different types of disfluencies. corrections. Table 1 gives a few examples. Inter- regnums are relatively easier to detect as they are often fixed phrases, e.g. “uh”, “you know”. On the other hand, reparandums are more difficult to detect in that they are in free form. As a result, most previous disfluency detection work focuses on detecting reparandums. Most work (Zayats and Ostendorf, 2018; Lou and Johnson, 2017; Wang et al., 2017; Jamshid Lou et al., 2018; Zayats and Ostendorf, 2019) on disflu- ency detection heavily relies on human-annotated corpora, which is scarce and expensive to obtain in practice. There have been several proposals to alle- viate this issue with, for instance, self-supervised learning …

Content truncated to 500 words