Multilingual Disfluency Removal using NMT

Multilingual Disfluency Removal using NMT Eunah Cho, Jan Niehues, Thanh-Le Ha, Alex Waibel Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe, Germany firstname.lastname@kit.edu Abstract In this paper, we investigate a multilingual approach for speech disfluency removal. A major challenge of this task comes from the costly nature of disfluency annotation. Moti- vated by the fact that speech disfluencies are commonly ob- served throughout different languages, we investigate the po- tential of multilingual disfluency modeling. We suggest that learning a joint representation of the disfluencies in multiple languages can be a promising solution to the data sparsity is- sue. In this work, we utilize a multilingual neural machine translation system, where a disfluent speech transcript is di- rectly transformed into a cleaned up text. Disfluency removal experiments on English and German speech transcripts show that multilingual disfluency model- ing outperforms the single language systems. In a follow- ing experiment, we show that the improvements are also observed in a downstream application using the disfluency- removed transcripts as input. 1. Introduction The challenges posed by spoken language translation (SLT) have received a great deal of research interest lately. One of these challenges is the handling of speech disfluencies. Un- like written language, spoken language contains disfluencies, such as repetitions, false starts, stutters, etc. Speech disflu- encies not only hinder the readability of speech transcripts but also harms the performance of subsequent applications of natural language processing (NLP), including machine trans- lation (MT). One challenge of speech disfluency modeling is the costly nature of disfluency-annotated data. Most state-of- the-art techniques for disfluency detection rely on supervised data, whose annotation process is expensive. Moreover, disfluency-annotated data is often limited to only certain lan- guages, which brings an even bigger challenge when model- ing the speech phenomenon for low resourced languages. Many of the speech disfluencies, such as stutters, false starts and repetitions, commonly occur in different lan- guages. This research, thus, is motivated by the idea that the data sparsity issue can be remedied if joint representations of speech disfluencies across languages are available. Recently, the characteristics of neural machine translation (NMT) uti- lizing continuous representations of the input sentences in- stead of a language-specific, fixed format have brought a lot of attention in building a model for multilingual inputs [1, 2]. The potential benefit from sharing semantic representations of the languages is a strong advantage of multilingual NMT. From this assessment, we expect that multilingual disfluency learning can be a solution to a task with the data sparsity is- sue. Utilizing the neural network (NN)-based system, we aim to obtain a joint representation of speech disfluencies across different languages. NMT offers a powerful framework where further speech- cleaning or reconstructing operations are allowed, such as reordering and replacement of words. As an initial work to apply the NMT framework for this task, we establish the ef- fectiveness of this framework on the disfluency removal task. We use disfluency-annotated English meeting data and German lecture data, and build three separated disfluency re- …

Content truncated to 500 words