LEXPLAIN: Improving Model Explanations via Lexicon Supervision

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 1–10 July 13-14, 2023 ©2023 Association for Computational Linguistics Including Facial Expressions in Contextual Embeddings for Sign Language Generation Carla Viegas1,2 and Mert Inan3 and Lorna Quandt4 and Malihe Alikhani 3 1Stella AI, Pittsburgh, USA 2Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA 3 Computer Science Department, School of Computing and Information, University of Pittsburgh, Pittsburgh, USA 4 Educational Neuroscience Program, Gallaudet University, Washington, D.C, USA Abstract State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing on man- ual signs, neglecting the affective, grammatical, and semantic functions of facial expressions. The purpose of this work is to augment se- mantic representation of sign language through grounding facial expressions. We study the effect of modeling the relationship between text, gloss, and facial expressions on the per- formance of the sign generation systems. In particular, we propose a Dual Encoder Trans- former able to generate manual signs as well as facial expressions by capturing the similarities and differences found in the text and sign gloss annotation. We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation. We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language. 1 Introduction Communication between the Deaf and Hard of Hearing (DHH) people and hearing non-signing people may be facilitated by emerging language technologies. DHH individuals are medically un- derserved worldwide (McKee et al., 2020; Masuku et al., 2021) due to the lack of doctors who can understand and use sign language. Also, educa- tional resources that are available in sign language are limited especially in STEM fields (Boyce et al., 2021; Lynn et al., 2020). Although the Americans with Disabilities Act (United States Department of Justice, 2010) requires government services, pub- lic accommodations, and commercial facilities to communicate effectively with DHH individuals, the reality is far from ideal. Sign language inter- preters are not always available, and communicat- ing through text is not always feasible as written languages are completely different from signed lan- guages. In contrast to Sign Language Recognition (SLR) which has been studied for several decades (Rast- goo et al., 2021) in the computer vision commu- nity (Yin et al., 2021), Sign Language Generation (SLG) is a more recent and less explored research topic (Quandt et al., 2021; Cox et al., 2002; Glauert et al., 2006). Missing a rich, grounded semantic representa- tion, the existing SLG frameworks are far from gen- erating understandable and natural sign language. Sign languages use spatiotemporal modalities and encode semantic information in manual signs and facial expressions. A major focus in SLG has been put on manual signs, neglecting the affective, gram- matical, and semantic roles of facial expressions. In this work, we bring insights from computational linguistics to study the role of and include facial expressions in …

Content truncated to 500 words