Building a conversation corpus by text derivation from “germ dialogs”

EAMT 2005 Conference Proceedings 27 Building a Conversation Corpus by Text Derivation from “Germ Dialogs” Naoki Asanoma, Setsuo Yamada, Osamu Furuse, and Masahiro Oku NTT Cyber Solutions Laboratories, NTT Corporation {asanoma.naoki, yamada.setsuo, furuse.osamu, oku.masahiro}@lab.ntt.co.jp Abstract. We propose a method for building a spoken-language text corpus for a spoken- language system. Conventional methods to build a new corpus include transcribing re- corded conversations, collecting text from existing documents, or writing original texts. However, these often have difficulties, such as insufficient corpus size and low cost effec- tiveness, when preparing the text data in the applied system’s domain. To address these is- sues, we have developed a method that uses “germ dialogs,” which are short-scripted dia- logs that enable writers to continue or replace them in a logical sequence that sounds natu- ral. This enables the corpus size to be proliferated in a cost-effective manner. Our results show that the proposed method can be used to create higher degree of adequateness for the system’s domain than conventional methods. The text data collected for the proposed method are used to generate a language model for our speech translation system between English and Japanese. 1. Introduction A text corpus plays a crucial role in many spo- ken-language applications, such as speech trans- lation and statistical natural language process- ing. The system’s accuracy often depends on whether we can accumulate a large amount and wide variety of text data containing frequent or domain-specific linguistic expressions. How- ever, there are fewer existing spoken-language corpora than there are written-language corpora. To make matters much more difficult, spoken- language corpora specific to the systems’ do- main is often unlikely to even exist. For these reasons, we must make an effort to build a spo- ken-language corpus in the system’s domain. Conventionally, a spoken-language corpus has been built using the following four methods: (a) Collecting text from existing documents : The text related to the system’s domain is copied from existing documents. Electronic data can be also used in some cases. (b) Transcribing recorded dialogs (Hirschman, 1992; Heeman & Allen, 1995; Takezawa, 1999; Allwood et al., 2000): A scripted, situational dialog is recorded. Then, the recorded dialogs are transcribed. (c) Storing keyboard chats (Kikui et al., 2003): Two participants chat through their keyboard terminals according to preferences or interests. The chat logs are stored as text data. (d) Writing imaginatively (Hirasawa et al., 2004): Given specific conversational scenarios, wri- ters imagine the following scenes and then cre- ate sentences that are likely to be uttered. If we can find a lot of text related to the sys- tem's domain, (a) is the most suitable method. However, most of the time very little text ex- ists. Additionally, copyright problems can arise. To avoid these problems, method (b) or (c) is usually used. Method (b) approximates the scenes to which the system will be actually applied and produces good quality text. For example, the CALLHOME corpus from the Linguistic Data Consortium was constructed using this approach (CALLHOME, 1996). However, using method …

Content truncated to 500 words