A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks

A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks Arif Khan1–3, Ingmar Steiner1,2, Yusuke Sugano4, Andreas Bulling1,5, Ross Macdonald6 1Multimodal Computing and Interaction, Saarland University, Germany, 2German Research Center for Artificial Intelligence (DFKI GmbH), Saarbr¨ucken, Germany, 3Saarbr¨ucken Graduate School of Computer Science, Germany, 4Osaka University, Japan 5Max Planck Institute for Informatics, Saarbr¨ucken, Germany 6University of Manchester, UK, {arifkhan,steiner}@coli.uni-saarland.de Abstract Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as close to the manual segmentation as possible. This corpus is an effort to capture the human segmentation behavior by recording experts performing a segmentation task. We believe that this data will enable us to highlight the important aspects of manual segmentation, which can be used in automatic segmentation to improve its accuracy. Keywords: eyetracking, gaze analysis, manual segmentation behavior 1. Introduction Speech segmentation is the process of splitting the acous- tic speech signal into distinct units by placing timestamped boundaries. This forms a crucial data processing step for phonetic analysis, as well as speech technology applica- tions such as text-to-speech synthesis and automatic speech recognition. The results and output quality depend on ac- curately segmented speech data. Speech segmentation can be done manually, using special- ized software, e.g., Praat (Boersma, 2001), Wavesurfer (Beskow and Sj¨olander, 2000), ELAN (Sloetjes and Wit- tenburg, 2008), and EMU (Winkelmann et al., 2017). In this workflow, a speech recording is displayed as a wave- form and/or spectrogram, and boundaries are inserted us- ing the mouse or keyboard (cf. Figure 1a). Short audio seg- ments can be played back to validate the boundary place- ment. This process is repeated until the whole audio file is segmented. Manual segmentation by experts is considered to produce the best phonetic segmentation one can achieve for any given data (Svendsen and Soong, 1987; Wesenick and Kipp, 1996). One reason for this is that they combine experience with multiple sources of information. However, there are some critical drawbacks of manual segmentation which make it impractical for large speech corpora. The first is that it is very laborious and time consuming; on av- erage, manual segmentation can take up to 30 s per phone (Leung and Zue, 1984; Stolcke et al., 2014) to segment. As a result, newly recorded speech data cannot be used quickly if manual segmentation is desired. Secondly, the exact placement of boundaries is subjective, and there may be disagreement between multiple experts. The second method of segmentation is doing it automati- cally, by training a model on the audio data, and then us- ing it to segment speech. In this method, the accuracy of the segmented speech directly depends on the quality of the trained model, which itself depends on the quality of train- ing data. Previous studies …

Content truncated to 500 words