Normalising Audio Transcriptions for Unwritten Languages

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 527–535, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Normalising Audio Transcriptions for Unwritten Languages Adel Foda∗ and Steven Bird† ∗Department of Mathematics and Statistics †Department of Computer Science and Software Engineering University of Melbourne, Victoria 3010, Australia Abstract The task of documenting the world’s lan- guages is a mainstream activity in linguis- tics which is yet to spill over into computa- tional linguistics. We propose a new task of transcription normalisation as an algo- rithmic method for speeding up the pro- cess of transcribing audio sources, leading to text collections of usable quality. We report on the application of sentence and word alignment algorithms to this task, before describing a new algorithm. All of the algorithms are evaluated over syn- thetic datasets. Although the results are nuanced, the transcription normalisation task is suggested as an NLP contribution to the grand challenge of documenting the world’s languages. 1 Introduction The majority of the world’s 6800 languages are rel- atively unstudied. Although some of the world’s languages have been carefully described and anal- ysed, most of them have not yet even been docu- mented. Such documentation consists of ‘compre- hensive and transparent records supporting wide ranging scientific investigations of the language’ (Woodbury, 2010). In this context then, it is striking that 50 years of research in computational linguistics have so far only touched about 1% of the world’s languages. In 100 years, 90% will be extinct or on the way out (Krauss, 2007). Ac- cordingly, we set ourselves the following question: what can computational linguistics offer to support the urgent task of documenting and analyzing the world’s endangered languages? There have been other recent efforts to address this question, fo- cussing on interlinear text (Xia and Lewis, 2007; Baldridge and Palmer, 2009). Our focus is dif- ferent, being concerned with creating the unanno- tated text that is presupposed by this earlier work. We also differentiate our work from more general computational support for documentary and de- scriptive linguistics, such as tools for transcribing audio or editing lexicons. Recently, Abney and Bird (2010) have proposed to incorporate machine translation (MT) into the workflow of language documentation. However, a significant challenge for this program is posed by the fact that the majority of the world’s languages are not written. How can NLP techniques be ap- plied to improve the speed and efficiency of audio transcription for unwritten languages? The task of transcription differs from transliter- ation (Knight and Graehl, 1998) in several ways. Transliteration is required in the context of ma- chine translation for dealing with proper names, which are a common source of out-of-vocabulary items. The goal is to make the words pronounce- able in a target language having a different inven- tory of sounds and syllables, and having different grapheme-to-phoneme rules. Since the source and target languages have established orthographies, the correct placement of word boundaries is never in question. Transcription, on the other hand, involves rep- …

Content truncated to 500 words