Efficient Disfluency Detection with Transition-based Parsing

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 495–503, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Efficient Disfluency Detection with Transition-based Parsing Shuangzhi Wu† , Dongdong Zhang‡ , Ming Zhou‡ , Tiejun Zhao† †Harbin Institute of Technology ‡Microsoft Research {v-shuawu, dozhang, mingzhou}@microsoft.com tjzhao@hit.edu.cn Abstract Automatic speech recognition (ASR) out- puts often contain various disfluencies. It is necessary to remove these disfluencies before processing downstream tasks. In this paper, an efficient disfluency detection approach based on right-to-left transition- based parsing is proposed, which can effi- ciently identify disfluencies and keep ASR outputs grammatical. Our method exploits a global view to capture long-range de- pendencies for disfluency detection by in- tegrating a rich set of syntactic and dis- fluency features with linear complexity. The experimental results show that our method outperforms state-of-the-art work and achieves a 85.1% f-score on the com- monly used English Switchboard test set. We also apply our method to in-house an- notated Chinese data and achieve a sig- nificantly higher f-score compared to the baseline of CRF-based approach. 1 Introduction With the development of the mobile internet, speech inputs have become more and more popu- lar in applications where automatic speech recog- nition (ASR) is the key component to convert speech into text. ASR outputs often contain var- ious disfluencies which create barriers to sub- sequent text processing tasks like parsing, ma- chine translation and summarization. Usually, disfluencies can be classified into uncompleted words, filled pauses (e.g. “uh”, “um”), discourse markers (e.g. “I mean”), editing terms (e.g. “you know”) and repairs. To identify and remove dis- fluencies, straightforward rules can be designed to tackle the former four classes of disfluencies since they often belong to a closed set. However, the repair type disfluency poses particularly more difficult problems as their form is more arbitrary. Typically, as shown in Figure 1, a repair disflu- ency type consists of a reparandum (“to Boston”) and a filled pause (“um”), followed by its repair (“to Denver”). This special structure of disfluency constraint, which exists in many languages such as English and Chinese, reflects the scenarios of spontaneous speech and conversation, where peo- ple often correct preceding words with following words when they find that the preceding words are wrong or improper. This procedure might be interrupted and inserted with filled pauses when people are thinking or hesitating. The challenges of detecting repair disfluencies are that reparan- dums vary in length, may occur everywhere, and are sometimes nested. I want a flight to Boston um to Denver FP RM RP correct Figure 1: A typical example of repair type disflu- ency consists of FP (Filled Pause), RM (Reparan- dum), and RP (Repair). The preceding RM is cor- rected by the following RP. There are many related works on disfluency detection, that mainly focus on detecting repair type of disfluencies. Straightforwardly, disflu- ency detection can be treated as a sequence la- beling problem and solved by …

Content truncated to 500 words