Lexicalized Phonotactic Word Segmentation

Proceedings of ACL-08: HLT, pages 130–138, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Lexicalized phonotactic word segmentation Margaret M. Fleck Department of Computer Science University of Illinois Urbana, IL 61801, USA mfleck@cs.uiuc.edu Abstract This paper presents a new unsupervised algo- rithm (WordEnds) for inferring word bound- aries from transcribed adult conversations. Phone ngrams before and after observed pauses are used to bootstrap a simple dis- criminative model of boundary marking. This fast algorithm delivers high performance even on morphologically complex words in English and Arabic, and promising results on accurate phonetic transcriptions with extensive pronun- ciation variation. Expanding training data be- yond the traditional miniature datasets pushes performance numbers well above those previ- ously reported. This suggests that WordEnds is a viable model of child language acquisition and might be useful in speech understanding. 1 Introduction Words are essential to most models of language and speech understanding. Word boundaries define the places at which speakers can fluently pause, and limit the application of most phonological rules. Words are a key constituent in structural analy- ses: the output of morphological rules and the con- stituents in syntactic parsing. Most speech recog- nizers are word-based. And, words are entrenched in the writing systems of many languages. Therefore, it is generally accepted that children learning their first language must learn how to seg- ment speech into a sequence of words. Similar, but more limited, learning occurs when adults hear speech containing unfamiliar words. These words must be accurately delimited, so that they can be added to the lexicon and nearby familiar words rec- ognized correctly. Current speech recognizers typi- cally misinterpret such speech. This paper will consider algorithms which seg- ment phonetically transcribed speech into words. For example, Figure 1 shows a transcribed phrase from the Buckeye corpus (Pitt et al., 2005; Pitt et al., 2007) and the automatically segmented output. Like almost all previous researchers, I use human- transcribed input to work around the limitations of current speech recognizers. In most available datasets, words are transcribed using standard dictionary pronunciations (hence- forth “dictionary transcriptions”). These transcrip- tions are approximately phonemic and, more impor- tantly, assign a constant form to each word. I will also use one dataset with accurate phonetic tran- scriptions, including natural variation in the pronun- ciation of words. Handling this variation is an im- portant step towards eventually using phone lattices or features produced by real speech recognizers. This paper will focus on segmentation of speech between adults. This is the primary input for speech recognizers. Moreover, understanding such speech is the end goal of child language acquisition. Models tested only on simplified child-directed speech are incomplete without an algorithm for upgrading the understander to handle normal adult speech. 2 The task in more detail This paper uses a simple model of the segmentation task, which matches prior work and the available datasets. Possible enhancements to the model are discussed at the end. 130 "all the kids in there # are people that have kids # …

Content truncated to 500 words