Towards Automatic Sign Translation

Towards Automatic Sign Translation Jie Yang, Jiang Gao, Ying Zhang, Alex Waibel Interactive Systems Laboratory Carnegie Mellon University Pittsburgh, PA 15213 USA {yang+,jgao,joy,waibel}@cs.cmu.edu ABSTRACT Signs are everywhere in our lives. They make our lives easier when we are familiar with them. But sometimes they also pose problems. For example, a tourist might not be able to understand signs in a foreign country. In this paper, we present our efforts towards automatic sign translation. We discuss methods for automatic sign detection. We describe sign translation using example based machine translation technology. We use a user- centered approach in developing an automatic sign translation system. The approach takes advantage of human intelligence in selecting an area of interest and domain for translation if needed. A user can determine which sign is to be translated if multiple signs have been detected within the image. The selected part of the image is then processed, recognized, and translated. We have developed a prototype system that can recognize Chinese signs input from a video camera which is a common gadget for a tourist, and translate them into English text or voice stream. Keywords Sign, sign detection, sign recognition, sign translation. 1. INTRODUCTION Languages play an important role in human communication. We communicate with people and information systems through diverse media in increasingly varied environments. One of those media is a sign. A sign is something that suggests the presence of a fact, condition, or quality. Signs are everywhere in our lives. They make our lives easier when we are familiar with them. But sometimes they also pose problems. For example, a tourist might not be able to understand signs in a foreign country. Unfamiliar language and environment make it difficult for international tourists to read signs, take a taxi, order food, and understand the comments of passersby. At the Interactive Systems Lab of Carnegie Mellon University, we are developing technologies for tourist applications [12]. The systems are equipped with a unique combination of sensors and software. The hardware includes computers, GPS receivers, lapel microphones and earphones, video cameras and head-mounted displays. This combination enables a multimodal interface to take advantage of speech and gesture inputs to provide assistance for tourists. The software supports natural language processing, speech recognition, machine translation, handwriting recognition and multimodal fusion. A vision module is trained to locate and read written language, is able to adapt to new environments, and is able to interpret intentions offered by the user, such as a spoken clarification or pointing gesture. In this paper, we present our efforts towards automatic sign translation. A system capable of sign detection and translation would benefit three types of individuals: tourists, the visually handicapped and military intelligence. Sign translation, in conjunction with spoken language translation, can help international tourists to overcome these barriers. Automatic sign recognition can help us to increase environmental awareness by effectively increasing our field of vision. It can also help blind people to extract information. A successful sign translation system relies on three key technologies: sign …

Content truncated to 500 words