What Comes First: Combining Motion Capture and Eye Tracking Data to Study the Order of Articulators in Constructed Action in Sign Language Narratives

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6003–6007 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6003 What Comes First: Combining Motion Capture and Eye Tracking Data to Study the Order of Articulators in Constructed Action in Sign Language Narratives Tommi Jantunen1, Anna Puupponen2, Birgitta Burger3 University of Jyväskylä, 1,2 Sign Language Centre & 3 Department of Music, Art and Culture Studies P.O. Box 35, FI-40014 University of Jyväskylä, Finland {tommi.j.jantunen, anna.m.puupponen, birgitta.burger}@jyu.fi Abstract We use synchronized 120 fps motion capture and 50 fps eye tracking data from two native signers to investigate the temporal order in which the dominant hand, the head, the chest and the eyes start producing overt constructed action from regular narration in seven short Finnish Sign Language stories. From the material, we derive a sample of ten instances of regular narration to overt constructed action transfers in ELAN which we then further process and analyze in Matlab. The results indicate that the temporal order of articulators shows both contextual and individual variation but that there are also repeated patterns which are similar across all the analyzed sequences and signers. Most notably, when the discourse strategy changes from regular narration to overt constructed action, the head and the eyes tend to take the leading role, and the chest and the dominant hand tend to start acting last. Consequences of the findings are discussed. Keywords: motion capture, eye tracking, sign language, constructed action, narration 1. Introduction Constructed action (CA) is an enactment-based discourse strategy in which signers (and speakers alike) use their hands and other parts of the body to show (as opposed to tell about) their interpretations of the actions, thoughts, feelings and speech of characters they are referring to in the discourse (Cormier, Smith & Sevcikova Sehyr, 2015; Ferrara & Hodge, 2018; Hodge & Cormier, 2019). CA forms a continuum with regular narration (RN, i.e. telling; Jantunen, 2017), and on the basis of the number of enacting articulators and the prominence of the character perspective adopted by the signer, CA can be divided into three degrees or subtypes: overt, reduced and subtle (Cormier, Smith & Sevcikova Sehyr, 2015). Overt CA is the strongest of the three types and, definitionally, it comprises only unconventional and gradient (i.e. non-lexical, often also referred to as gestural) elements used fully from a character perspective. At the other end of the continuum, RN comprises only highly conventional and discrete (i.e. lexical or partly-lexical) units that are produced fully from the perspective of the narrator. In this paper we use motion capture (MoCap) and eye tracking (ET) technology to investigate the temporal order in which the dominant hand, the head, the chest and the eyes start producing overt CA from RN in short Finnish Sign Language (FinSL) stories. Previous work on the topic does not exist, but the work on CA in general suggests that there are two schools of thought as regards the articulatory order: some studies imply that …

Content truncated to 500 words