Hierarchical Latent Words Language Models for Robust Modeling to Out-Of Domain Tasks

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1896–1901, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Hierarchical Latent Words Language Models for Robust Modeling to Out-Of Domain Tasks Ryo Masumura†‡, Taichi Asami†, Takanobu Oba†, Hirokazu Masataki†, Sumitaka Sakauchi†, Akinori Ito‡ † NTT Media Intelligence Laboratories, NTT Corporation, Japan ‡ Graduate School of Engineering, Tohoku University, Japan † {masumura.ryo, asami.taichi, oba.takanobu, masataki.hirokazu, sakauchi.sumitaka} @lab.ntt.co.jp, ‡ aito@spcom.ecei.tohoku.ac.jp Abstract This paper focuses on language modeling with adequate robustness to support differ- ent domain tasks. To this end, we propose a hierarchical latent word language model (h-LWLM). The proposed model can be regarded as a generalized form of the stan- dard LWLMs. The key advance is in- troducing a multiple latent variable space with hierarchical structure. The structure can flexibly take account of linguistic phe- nomena not present in the training data. This paper details the definition as well as a training method based on layer-wise inference and a practical usage in natural language processing tasks with an approx- imation technique. Experiments on speech recognition show the effectiveness of h- LWLM in out-of domain tasks. 1 Introduction Language models (LMs) are essential for auto- matic speech recognition or statistical machine translation (Rosenfeld, 2000). The performance of LMs strongly depends on quality and quantity of their training data. Superior performance is usu- ally obtained by using enormous domain-matched training data sets to construct LMs (Brants et al., 2007). Unfortunately, in many cases, large amounts of domain-matched training data sets are not available. Therefore, LM technology that can robustly work for domains that differ from that of the training data is needed (Goodman, 2001). For robust language modeling, several tech- nologies have been proposed. Fundamental tech- niques are smoothing (Chen and Goodman, 1999) and clustering (Brown et al., 1992). Other solu- tions are Bayesian modeling (Teh, 2006) and en- semble modeling (Xu and Jelinek, 2004; Emami and Jelinek, 2005). Moreover, continuous rep- resentation of words in neural network LMs can also support robust modeling (Bengio et al., 2003; Mikolov et al., 2010). However, previous works are focused on maximizing performance in the same domain as that of the training data. In other words, it is uncertain that these technologies ro- bustly support out-of domain tasks. In contrast, latent words LMs (LWLMs) (De- schacht et al., 2012) are clearly effective for out- of domain tasks. We employed the LWLM to speech recognition and the resulting performance was significantly superior in out-of domain tasks while the performance was comparable in domain- matched task to conventional LMs (Masumura et al., 2013a; Masumura et al., 2013b). LWLMs are generative models that employ a latent word space. The latent space can flexibly take into ac- count relationships between words and the model- ing helps to efficiently increase the robustness to out-of domain tasks (Sec. 2). In this paper, we focus on LWLMs and aim to make them more flexible for greater robustness to out-of domain tasks. To this end, this paper takes note …

Content truncated to 500 words