On the Interdependence of Language and Perception

ON THE INTERDEPENDENCE OF LANGUAGE AND PERCEPTION* David L. Waltz Coordinated Science Laboratory University of Illinois at Urbana/Champaign ABSTRACT It is argued that without a connection to the real world via perception, a language system can- not know what it is talking about. Similarly, a perceptual system must have ways of expressing its outputs via a language (spoken, written, gestural or other). The relationship between perception and language is explored, with special attention to the implications of results in language re- search for our models of vision systems, and vice- versa. It is suggested that early language learn- ing is an especially fertile area for this explo- ration. Within this area, we argue that perceptual data is conceptualized prior to language acquisition according to largely innate strategies, that this conceptualization is in terms of an internal, non- ambiguous "language," that language production from its beginnings to adulthood is a projection of the internal language which selects and high- lights the most important portions of internal concepts, and that schemata produced in the sensory/motor world are evolved into schemata to describe abstract worlds. Examples are provided which stress the important of "gestalt" (figure- ground) relationships and projection (3-D to 2-1/2 or 2-D, conceptual to linguistic, and linguistic to conceptual); finally mechanisms for an integrated vision-language system are proposed, and some preliminary results are described. Introduction perception i. obs.: CONSCIOUSNESS 2a: a result of perceiving: OBSERVATION b: a mental image: CONCEPT 3a: awareness of the elements of environment through physical sensation <color ~> b: physical sensation interpreted in the light of experience 4a: direct or intuitive cognition: INSIGHT b: a capacity for comprehension syn see DISCERNMENT (Webster's Seventh New Collegiate Dictionary) *This work was supported by the Office of Naval Research under Contract ONR-NO0014-75-C-0612. tWhile I intend perception to refer in the human examples to all the senses: vision, hearing, touch, smell, taste, and motor sense, in the case of computers, only vision has been explored in more than a cursory manner. Language understanding in its deepest sense is not possible without direct experience of its real world correlates. I believe that it is no accident that the same word can refer both to sensory awareness and to comprehension. Nearly all efforts in language processing, both in artificial intelligence and linguistics, have concentrated on transforming strings of words into trees or other structures of words (sometimes surface words, sometimes "primitive" words) or conversely, on producing strings of words from these structures. Few researchers have even recognized the importance of interfacing language and vision systems,t let alone uniting the two lines of research. (Exceptions include [Minsky 1975], [Woods 1978], [Miller & Johnson-Laird [1976], [Schank & Selfridge !977], [Pylyshyn 1977 a & b], [Clark 1973]). At this time in history, AI vision and natural language researchers have little to say to each other; most of the work which treats language and perception together would I think be considered to lie in the realms of philosophy or psychology. Moreover, the areas of language processing which …

Content truncated to 500 words