A Comparative Analysis of Crowdsourced Natural Language Corpora for Spoken Dialog Systems

A Comparative Analysis of Crowdsourced Natural Language Corpora for Spoken Dialog Systems Abstract Recent spoken dialog systems have been able to recognize freely spoken user input in restricted domains thanks to statistical methods in the automatic speech recognition. These methods require a high number of natural language utterances to train the speech recogni- tion engine and to assess the quality of the system. Since human speech offers many variants associated with a single intent, a high number of user utterances have to be elicited. Developers are therefore turning to crowdsourcing to collect this data. This paper compares three different methods to elicit multiple utterances for given semantics via crowd sourcing, namely with pictures, with text and with semantic entities. Specifically, we compare the methods with regard to the number of valid data and linguistic variance, whereby a quantitative and qualitative approach is proposed. In our study, the method with text led to a high variance in the utterances and a relatively low rate of invalid data. Keywords: natural language data, crowdsourcing, elicitation methods 1. Introduction Early spoken dialog systems (SDS) supported only a finite set of speech commands. Thanks to statistical methods in the automatic speech recognition (ASR) recent systems such as Apple’s Siri1 have been able to recognize freely spoken user input in restricted domains. These methods require a high number of spoken utter- ances to train the speech recognition engine. In order to train the speech recognition engine and to assess the quality of a SDS which is able to interpret any spoken user utterance associated with a given intent, it is necessary to collect data that reflects real usage. The elicitation of real user data is a challenge within the development process of such systems. From a commercial point of view, data collection for SDS should meet the following require- ments: - Allow for fast elicitation - Apply to different intents and domains - Reflect representative user input - Include a high rate of usable data - Cover high linguistic variance. Since development cycles of SDS are normally short and much data is needed, developers are turning to crowdsourcing (Eskenazi et al., 2013). Our goal is to investigate the use of crowdsourcing methods with regard to the requirements above. In order to collect real system interaction data, one should choose elicitation methods which do not bias the participants. However, a high number of valid utterances should be generated. In this paper, we collected natural language speech utter- ances via crowdsourcing with the goal of comparing three different elicitation methods. The participants were asked 1 http://www.apple.com/ios/siri/ to perform tasks such as entering the address into the navigation application. The elicitation methods differed in the way how the tasks were presented to the participants, by means of: - Pictures - Semantic entities - Text. Specifically, we compare the methods with regard to the number of valid data and linguistic variance, whereby a quantitative and qualitative analysis approach is pro- posed. The focus of the data we collected was on …

Content truncated to 500 words