PROPICTO: Developing Speech-to-Pictograph Translation Systems to Enhance Communication Accessibility

PROPICTO: Developing Speech-to-Pictograph Translation Systems to Enhance Communication Accessibility Luc´ıa Ormaechea1,2, Pierrette Bouillon1, Maximin Coavoux2, Emmanuelle Esperanc¸a-Rodier2, Johanna Gerlach1, Jerˆome Goulian2, Benjamin Lecouteux2, C´ecile Macaire2, Jonathan Mutal1, Magali Norr´e1, Adrien Pupier2, and Didier Schwab2 1 TIM/FTI, University of Geneva, 1205 Geneva – Switzerland {firstName.lastName}@unige.ch 2 Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble – France {firstName.lastName}@univ-grenoble-alpes.fr Abstract PROPICTO is a project funded by the French National Research Agency and the Swiss National Science Foundation, that aims at creating Speech-to-Pictograph translation systems, with a special focus on French as an input language. By de- veloping such technologies, we intend to enhance communication access for non- French speaking patients and people with cognitive impairments. 1 Introduction Alternative and augmentative communication (AAC) devices have taken an increasingly impor- tant role among people with disabilities and their relatives. However, usage of these technologies (i.e., communication boards or electronic media) may be cumbersome (Vaschalde et al., 2018). To surmount this problem, we argue that Speech-to- Pictograph (S2P) translation systems can be help- ful for AAC users. In addition, we believe that they can improve the accessibility of health services for patients not speaking the local language. Develop- ing such tools requires in-depth research on several areas of natural language processing (NLP). In this paper, we present a research project aimed at cre- ating systems that automatically translate spoken French into pictographs. Launched in early 2021, PROPICTO1 (the acronym stands for PROjecting spoken language into PICTOgraphs) is a French-Swiss four-year project, funded by both the French National Re- © 2023 The authors. This article is licensed under a Cre- ative Commons 4.0 licence, no derivative works, attribu- tion, CC-BY-ND. 1 https://www.propicto.unige.ch/ search Agency2 and the Swiss National Science Foundation.3 It is conducted as a collaboration be- tween the Department of Translation Technology at the University of Geneva and the Study Group for Machine Translation and Automated Process- ing of Languages and Speech, affiliated to the Grenoble Informatics Laboratory. For the purpose of this project, we will examine several NLP-related areas; namely, speech recog- nition, syntactic parsing, sentence simplification as well as pictograph generation. By integrating this series of tasks into different workflows (de- pending on the target scenario), we propose novel cross-modal machine translation systems that con- vert spoken language into pictographic units. Us- ing this approach, we intend to tackle societal and communicative needs in the fields of: (1) disabil- ity, where an individual seeks to communicate with a person having a cognitive disorder, and (2) med- ical settings, where a language barrier exists be- tween patient and practitioner. 2 Architectural Overview PROPICTO aims at improving the usability of AAC devices by leveraging NLP-based solu- tions for greater accessibility. We will de- sign new methods and corpora so as to en- able spoken utterances to be transcribed directly into sequences of pictographs, either general- purpose like ARASAAC,4 or specific-purpose (i.e., Sant´eBD5 for health-related concepts). The project will face two major challenges: • The scarcity of parallel speech-pictographs corpora, which constitutes a high hurdle to …

Content truncated to 500 words