Learning Spoken Language Representations with Neural Lattice Language Modeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3764–3769 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3764 Learning Spoken Language Representations with Neural Lattice Language Modeling Chao-Wei Huang Yun-Nung (Vivian) Chen Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan r07922069@ntu.edu.tw y.v.chen@ieee.org Abstract Pre-trained language models have achieved huge improvement on many NLP tasks. How- ever, these methods are usually designed for written text, so they do not consider the prop- erties of spoken language. Therefore, this paper aims at generalizing the idea of lan- guage model pre-training to lattices generated by recognition systems. We propose a frame- work that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. The pro- posed two-stage pre-training approach reduces the demands of speech data and has better ef- ficiency. Experiments on intent detection and dialogue act recognition datasets demonstrate that our proposed method consistently outper- forms strong baselines when evaluated on spo- ken inputs.1 1 Introduction The task of spoken language understanding (SLU) aims at extracting useful information from spoken utterances. Typically, SLU can be decomposed with a two-stage method: 1) an accurate automatic speech recognition (ASR) system transcribes the input speech into texts, and then 2) language under- standing techniques are applied to the transcribed texts. These two modules can be developed sepa- rately, so most prior work developed the backend language understanding systems based on manual transcripts (Yao et al., 2014; Guo et al., 2014; Mes- nil et al., 2014; Goo et al., 2018). Despite the simplicity of the two-stage method, prior work showed that a tighter integration be- tween two components can lead to better perfor- mance. Researchers have extended the ASR 1-best results to n-best lists or word confusion networks in order to preserve the ambiguity of the transcripts. 1The scource code is available at: https://github. com/MiuLab/Lattice-ELMo. Airfare 0.3 fair 1.0 to, 1.0 LA 1.0 Figure 1: Illustration of a lattice. (Tur et al., 2002; Hakkani-T¨ur et al., 2006; Hender- son et al., 2012; T¨ur et al., 2013; Masumura et al., 2018). Another line of research focused on using lattices produced by ASR systems. Lattices are directed acyclic graphs (DAGs) that represent mul- tiple recognition hypotheses. An example of ASR lattice is shown in Figure 1. Ladhak et al. (2016) in- troduced LatticeRNN, a variant of recurrent neural networks (RNNs) that generalize RNNs to lattice- structured inputs in order to improve SLU. Zhang and Yang (2018) proposed a similar idea for Chi- nese name entity recognition. Sperber et al. (2019); Xiao et al. (2019); Zhang et al. (2019) proposed ex- tensions to enable the transformer model (Vaswani et al., 2017) to consume lattice inputs for machine translation. Huang and Chen (2019) proposed to adapt the transformer model originally pre-trained on written texts to consume lattices in order to improve SLU performance. Buckman and Neu- big (2018) also found that utilizing lattices that represent multiple granularities of sentences can improve language modeling. With recent …

Content truncated to 500 words