Operational Assessment of Keyword Search on Oral History

Operational Assessment of Keyword Search on Oral History Elizabeth Salesky, Jessica Ray, and Wade Shen MIT Lincoln Laboratory Lexington, Massachusetts, USA {elizabeth.salesky, jessica.ray, swade}@ll.mit.edu Abstract This project assesses the resources necessary to make oral history searchable by means of automatic speech recognition (ASR). There are many inherent challenges in applying ASR to conversational speech: smaller training set sizes and varying demographics, among others. We assess the impact of dataset size, word error rate and term-weighted value on human search capability through an information retrieval task on Mechanical Turk. We use English oral history data collected by StoryCorps, a national organization that provides all people with the opportunity to record, share and preserve their stories, and control for a variety of demographics including age, gender, birthplace, and dialect on four different training set sizes. We show comparable search performance using a standard speech recognition system as with hand-transcribed data, which is promising for increased accessibility of conversational speech and oral history archives. Keywords: Keyword Search, Oral History, Mechanical Turk 1. Introduction Oral history has been specifically recorded for the purpose of making history available to future generations. While digitizing such archives has recently gained momentum, these recordings remain largely inaccessible without the ability to index and retrieve relevant items. Though some projects, like the Shoah Visual History Foundation (VHF) have the resources to transcribe their collections by hand (Byrne et al., 2004), this is typically not the case. While there have been many recent advances in the performance of automatic speech recognition (ASR) on spontaneous conversational speech both in and out of this domain, it re- mains to quantify the resources and techniques necessary for effective search performance. StoryCorps has recorded more than 56,000 interviews with over 90,000 participants to date, made available through the American Folklife Center within the Library of Congress and selectively online (Dave Isay, 2015). Search is cur- rently only available for metadata, making it difficult to find interviews discussing particular topics. We leverage stan- dard HMM/GMM-based ASR systems and keyword search (KWS) to make the content more accessible. Training usable ASR systems requires transcribed data: hundreds to thousands of hours for English. Most oral his- tory archives are smaller than this by an order of magnitude. We use recordings from the StoryCorps Griot Initiative, which contains 164 hours of audio, and compare results across varying dataset sizes. We compare standard auto- matic metrics for assessing search performance with actual human search capability as measured on Amazon Mechan- ical Turk (MTurk) in an information retrieval (IR) task. On the human task, we compare MTurk worker (Turker) per- formance on generated transcripts against reference tran- scripts and a metadata-only search condition. We assess accessibility by measuring the number of searches and time taken compared to the bail out point for an average subject. This work is sponsored by the Office of the Director of Na- tional Intelligence (ODNI) under Air Force Contract FA8721-05- C-0002, IARPA-BAA-11-02. Opinions, interpretations, conclu- sions and recommendations are those of the authors and are …

Content truncated to 500 words