The ILMT-s2s Corpus ― A Multimodal Interlingual Map Task Corpus

The ILMT-s2s Corpus — A Multimodal Interlingual Map Task Corpus Hayakawa Akira†, Saturnino Luz‡, Loredana Cerrato†, Nick Campbell† †ADAPT Centre, School of Computer Science and Statistics, Trinity College Dublin, Ireland ‡Usher Institute of Population Health Sciences & Informatics, University of Edinburgh, UK †{campbeak, cerratol, nick}@tcd.ie, ‡S.Luz@ed.ac.uk Abstract This paper presents the multimodal Interlingual Map Task Corpus (ILMT-s2s corpus) collected at Trinity College Dublin, and discuss some of the issues related to the collection and analysis of the data. The corpus design is inspired by the HCRC Map Task Corpus which was initially designed to support the investigation of linguistic phenomena, and has been the focus of a variety of studies of communicative behaviour. The simplicity of the task, and the complexity of phenomena it can elicit, make the map task an ideal object of study. Although there are studies that used replications of the map task to investigate communication in computer mediated tasks, this ILMT-s2s corpus is, to the best of our knowledge, the first investigation of communicative behaviour in the presence of three additional “filters”: Automatic Speech Recognition (ASR), Machine Translation (MT) and Text To Speech (TTS) synthesis, where the instruction giver and the instruction follower speak different languages. This paper details the data collection setup and completed annotation of the ILMT-s2s corpus, and outlines preliminary results obtained from the data. Keywords: interlingual map task, computer mediated communication, natural task oriented dialogue 1. Introduction The aim of this paper is to present the multimodal Inter- lingual Map Task Corpus (ILMT-s2s corpus) (Hayakawa et al., 2015) collected at Trinity College Dublin, and discuss some of the issues related to the collection and analysis of the data. Fifteen dialogues were collected with the map task elicita- tion technique in a setting where the interaction is mediated by a speech-to-speech (S2S) translation system. The sys- tem was implemented using off-the-shelf technology to en- able speakers of different languages to communicate with each other (remotely, over the network) in their native lan- guages. This prototype speech-to-speech translation sys- tem (ILMT-s2s system) adds three elements: Automatic Speech Recognition (ASR), Machine Translation (MT) and Text To Speech (TTS) synthesis into the communication — we call these “filters”. Our main interest is to understand how such filters affect the subjects in terms of cognitive load, adaptation of communicative acts to the technology, prosodic alignment and repair strategies, and other factors that might have implications for the design of dialogue sys- tems and, in particular, S2S translation systems. To this end we have collected a variety of synchronised and finely time-stamped data streams, including: high quality audio of the subjects’ utterances, video and ASR, MT, TTS events. In addition, heart rate, skin conductance, blood vol- ume pressure and electroencephelography (EEG) signals of one subject in each dialogue has also been recorded during the whole interaction with biosignal monitoring devices. So far the corpus has been used to investigate two speech related interaction phenomena observed in a map naviga- tion setting where the collaborating subjects speak different languages …

Content truncated to 500 words