Improving Spoken Language Understanding by Wisdom of Crowds

Proceedings of the 28th International Conference on Computational Linguistics, pages 2606–2612 Barcelona, Spain (Online), December 8-13, 2020 2606 Improving Spoken Language Understanding by Wisdom of Crowds Koichiro Yoshino2,1,3, Kana Ikeuchi1, Katsuhito Sudoh1,3 and Satoshi Nakamura1,3 1Nara Institute of Science and Technology (NAIST) 2RIKEN Robotics Project 3RIKEN Center for Advanced Intelligence Project (AIP) koichiro.yoshino@riken.jp, {sudoh, s-nakamura}@is.naist.jp Abstract Spoken language understanding (SLU), which converts user requests in natural language to machine-interpretable expressions, is becoming an essential task. The lack of training data is an important problem, especially for new system tasks, because existing SLU systems are based on statistical approaches. In this paper, we proposed to use two sources of the “wisdom of crowds,” crowdsourcing and knowledge community website, for improving the SLU system. We firstly collected paraphrasing variations for new system tasks through crowdsourcing as seed data, and then augmented them using similar questions from a knowledge community website. We inves- tigated the effects of the proposed data augmentation method in SLU task, even with small seed data. In particular, the proposed architecture augmented more than 120,000 samples to improve SLU accuracies. 1 Introduction Recent advances in speech applications running on smartphones and smart speakers increase the impor- tance of spoken language understanding (SLU). SLU is a task to predict an appropriate system function with its arguments, given a user request written or spoken in natural language. Various SLU benchmarks have been proposed: Air Travel Information Services (ATIS) (Dahl et al., 1994), restaurant information navigation (Williams et al., 2014), and other speech applications (Hori et al., 2019). Adaptation of SLU to newly defined tasks is an important problem (Henderson et al., 2014). The num- ber of training data directly affects the SLU accuracy because most of the recent SLU systems are based on statistical machine learning approaches. Some existing work tackled this problem based on transfer learning (Wu et al., 2019), which uses a pre-trained model on different domain data. However, it is still challenging to make the SLU accurate with no or fewer data. Data augmentation has been applied to solve this problem, which generates pseudo training samples (Hou et al., 2018; Yoo et al., 2019). How- ever, such methods often generate unnatural training samples that will decrease SLU accuracy. Another problem is the ambiguity of user utterances; it is difficult to generate such ambiguous examples with generative approaches. Some existing work tackled this problem by using paraphrasing models (Saha et al., 2018; Ray et al., 2018). On the other hand, collecting text data from the Web is a widely used approach to building language models of automatic speech recognition systems (Bulyko et al., 2003; Sarikaya et al., 2005; Ng et al., 2005; Tsiartas et al., 2010). Web texts are expected to be more natural than generated pseudo sentences because users handcraft most of them. However, Web texts contain diverse domain texts; thus, we need some criteria to select appropriate texts to be used for the augmented training data. Test-set perplexity (Misu and Kawahara, 2006) or semantic similarity (Hakkani-Tur and …

Content truncated to 500 words