Improving Disfluency Detection by Self-Training a Self-Attentive Model

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3754–3763 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3754 Improving Disfluency Detection by Self-Training a Self-Attentive Model Paria Jamshid Lou1 and Mark Johnson2,1 1Department of Computing, Macquarie University 2Oracle Digital Assistant, Oracle Corporation 1paria.jamshid-lou@hdr.mq.edu.au 2mark.mj.johnson@oracle.com Abstract Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detec- tion in speech transcripts. Since the contex- tualized word embeddings are pre-trained on a large amount of unlabeled data, using addi- tional unlabeled data to train a neural model might seem redundant. However, we show that self-training — a semi-supervised tech- nique for incorporating unlabeled data — sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word repre- sentations. We also show that ensembling self- trained parsers provides further gains for dis- fluency detection. 1 Introduction Speech introduces challenges that do not appear in written text, such as the presence of disflu- encies. Disfluency refers to any interruptions in the normal flow of speech, including false starts, corrections, repetitions and filled pauses. Shriberg (1994) defines three distinct parts of a speech dis- fluency, referred to as the reparandum, the inter- regnum and the repair. As illustrated in the exam- ple below, the reparandum The first kind of inva- sion of is the part of the utterance that is replaced or repaired, the interregnum uh I mean (which consists of a filled pause uh and a discourse marker I mean) is an optional part of the disfluency, and the repair the first type of privacy replaces the reparandum. The fluent version is obtained by re- moving the reparandum and the interregnum. reparandum � �� � The first kind of invasion of �interregnum �� � uh I mean �the first type of privacy �� � repair seemed invaded to me This paper will focus on joint disfluency de- tection and constituency parsing of transcribed speech. In the Switchboard treebank corpus (God- frey and Holliman, 1993; Marcus et al., 1999), which is a standard corpus for parsing studies on conversational speech, the reparanda, filled pauses and discourse markers are dominated by EDITED, INTJ and PRN nodes, respectively (see Figure 1). Filled pauses and discourse markers be- long to a finite set of words and phrases, so INTJ and PRN nodes are trivial to detect (Johnson and Charniak, 2004). Detecting EDITED nodes, how- ever, is challenging and is the main focus of dis- fluency detection models. S EDITED NP NP DT The JJ first NN kind PP IN of NP NP NN invasion PP IN of INTJ UH uh PRN S NP PRP I VP VBP mean NP NP DT the JJ first NN type PP IN of NP NN privacy VP VBD seemed VP VBN invaded PP IN to NP PRP me Figure 1: A parse tree from the Switchboard corpus, where reparandum The first kind of …

Content truncated to 500 words