Automated Generation of Storytelling Vocabulary from Photographs for use in AAC

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1353–1364 August 1–6, 2021. ©2021 Association for Computational Linguistics 1353 Automated Generation of Storytelling Vocabulary from Photographs for Use in AAC Mauricio Fontana de Vargas and Karyn Moffatt School of Information Studies McGill University, Montreal, Canada mauricio.fontanadevargas@mail.mcgill.ca, karyn.moffatt@mcgill.ca Abstract Research on the application of NLP in symbol- based Augmentative and Alternative Commu- nication (AAC) tools for improving social in- teraction support is scarce. We contribute a novel method for generating context-related vocabulary from photographs of personally rel- evant events aimed at supporting people with language impairments in recounting their past experiences. Performance was calculated with information retrieval concepts on the relevance of vocabulary generated for communicating a corpus of 9730 narrative phrases about events depicted in 1946 photographs. In comparison to a baseline generation composed of frequent English words, our method generated vocab- ulary with a 4.6 gain in mean average preci- sion, regardless of the level of contextual infor- mation in the input photographs, and 6.9 for photographs in which contextual information was extracted correctly. We conclude by dis- cussing how our fndings provide insights for system optimization and usage. 1 Introduction Augmentative and Alternative Communication (AAC) tools can enhance communication for non- speaking individuals, thus offering improved social interaction and independence. Well established NLP techniques, such as spell check and word prediction, support those with primarily physical barriers to communication (e.g., adults with ALS) to compose complex and nuanced sentences in orthographic-based systems more effciently. How- ever, those with developmental disabilities (e.g., autism spectrum disorder, ASD) or lexical and se- mantic processing impairments that limit their abil- ity to spell out words (e.g., adults with aphasia1) must usually rely on less expressive symbol-based systems, for which those techniques offer little sup- 1a language disorder mostly often caused by a stroke. port due to unique characteristics of communica- tion with these systems. Users of symbol-based AAC typically do not construct full, grammatically correct sentences, complete with prepositions and infections, but rather often only need a few key content words (i.e., nouns, adjectives, verbs)—appearing at any part of the sentence— to supplement other forms of communication, including preserved speech, ges- tures, or drawings. Such scattered use of vocab- ulary hinders the typical statistical prediction ap- proach, which relies on patterns learnt from a large training corpus. Nonetheless, there is much opportunity for im- proving symbol-based AAC, which is often aban- doned because it offers too little communication support relative to the effort required to learn and use (Moffatt et al., 2017). Figure 1: An AAC app design demonstrating how context-related vocabulary generated by our method might be presented for use in subsequent conversations. As in many non-orthographic AACs, vocabulary is rep- resented by images that reproduce computer generated speech when selected; however, unlike the status quo, this design eliminates navigation across complicated hi- erarchies and the need for pre-programming. 1354 Selecting and organizing vocabularies able to attend …

Content truncated to 500 words