Toward Communicating Simple Sentences Using Pictorial Representations

Toward Communicating Simple Sentences Using Pictorial Representations Rada Mihalcea and Ben Leong Department of Computer Science University of North Texas {rada,cl0198}@cs.unt.edu Abstract This paper evaluates the hypothesis that pictorial representations can be used to ef- fectively convey simple sentences across language barriers. Comparative evalu- ations show that a considerable amount of understanding can be achieved using visual descriptions of information, with evaluation figures within a comparable range of those obtained with linguistic representations produced by an automatic machine translation system. 1 Introduction Universal communication represents one of the long-standing goals of the humanity – borderless communication among people, regardless of the lan- guage they speak. According to recent studies (Eth, 2005), (Gibbs, 2002) there are about 7,000 lan- guages spoken worldwide. From these, only about 15–20 languages can currently take advantage of the benefits provided by machine translation, and even for these languages, the automatically pro- duced translations are not error free and their quality lags behind the human expectations. In this paper, we investigate a new paradigm for translation: translation through pictures, as op- posed to translation through words, as a means for producing universal representations of information that can be effectively conveyed across language barriers. Regardless of the language they speak, peo- ple share almost the same ability to understand the content of pictures. For instance, speakers of dif- ferent languages have a different way of referring to the concept of apple, as illustrated in Figure 1(a). Instead, a picture can be understood by all people in the same way, replacing the multitude of linguistic descriptions with one, virtually universal represen- tation (Figure 1(b)). (Japanese) (Arabic) (Chinese) pomme (French) manzana (Spanish) mar (Romanian) apple (English) (a) linguistic representations (b) pictorial representation elma (Turkish) apel (Indonesian) alma (Hungarian) Figure 1: Linguistic and visual representations for the concept “apple”. In addition to enabling communication across lan- guages, the ability to encode information using pic- torial representations has other benefits, such as lan- guage learning for children or for those who study a foreign language, communication to and from prelit- erate or non-literate people, or language understand- ing for people with language disorders. This paper describes a system for the automatic generation of pictorial translations for simple sen- tences, and evaluates the hypothesis that such picto- rial descriptions can be understood independent of language-specific representations. An example of the pictorial translations that we target is shown in Figure 2(a). There are of course limitations inherent to the use of visual representations for the purpose of commu- 119 Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pages 119-127, Cambridge, August 2006. ©2006 The Association for Machine Translation in the Americas nication. First, there are complex informations that cannot be conveyed through pictures, as in e.g. “An inhaled form of insulin won federal approval yester- day,” which require the more advanced representa- tions that can only be encoded in language. Second, there is a large number of concepts that have a level of abstraction that prohibits a …

Content truncated to 500 words