Increasing our Ignorance’ of Language: Identifying Language Structure in an Unknown ‘Signal’

In: Proceedings of CoNLL-2000 and LLL-2000, pages 25-30, Lisbon, Portugal, 2000. Increasing our Ignorance of Language: Identifying Language Structure in an Unknown 'Signal' John Elliott and Eric Atwell and Bill Whyte Centre for Computer Analysis of Language and Speech, School of Computer Studies University of Leeds, Leeds, Yorkshire, LS2 9JT England {jre, eric, billw}¢scs.leeds.ac.uk Abstract This paper describes algorithms and software developed to characterise and detect generic intelligent language-like features in an input signal, using natural language learning tech- niques: looking for characteristic statistical "language-signatures" in test corpora. As a first step towards such species-independent language-detection, we present a suite of pro- grams to analyse digital representations of a range of data, and use the results to extrap- olate whether or not there are language-like structures which distinguish this data from other sources, such as music, images, and white noise. Outside our own immediate NLP sphere, generic communication techniques are of par- ticular interest in the astronautical community, where two sessions are dedicated to SETI at their annual International conference with top- ics ranging from detecting ET technology to the ethics and logistics of message construction (E1- liott and Atwell, 1999; Ollongren, 2000; Vakoch, 2000). 1 Introduction A useful thought experiment is to imagine eavesdropping on a signal from outer space. How can you decide that it is a message be- tween intelligent life forms? We need a 'lan- guage detector': or, to put it more accu- rately, something that separates language from non-language. But what is special about the language signal that separates it from non- language? Is it, indeed, separable? The problem goal is to separate language from non-language without dialogue, and learn something about the structure of language in the passing. The language may not be human (animals, aliens, computers...), the perceptual space can be unknown, and we cannot assume human language structure but must begin some- where. We need to approach the language signal from a naive viewpoint, in effect, increasing our ignorance and assuming as little as possible. Given this standpoint, an informal descrip- tion of 'language' might include that it: • has structure at several interrelated levels • is not random • has grammar • has letters/characters, words, phrases and sentences • has parts of speech • is recursive • has a theme with variations • is aperiodic but evolving • is generative • has transformation rules • is designed for communication • has Zipfian type-token distributions at sev- eral levels Language as a 'signal' • has some signalling elements (a 'script') • has a hierarchy of signalling elements? ('Words', 'phrases' etc.) • is serial? • is correlated across a distance of several sig- nalling elements applying at various levels in the hierarchy • is usually not truly periodic • is quasi-stationary? • is non-ergodic? We assume that a language-like signal will be encoded symbolically, i.e. with some kind of character-stream. Our language-detection al- gorithm for symbolic input uses a number of 25 statistical clues such as entropy, "chunking" to find character bit-length and …

Content truncated to 500 words