SCALE: A Scalable Language Engineering Toolkit

SCALE: A Scalable Language Engineering Toolkit Joris Pelemans1, Lyan Verwimp1, Kris Demuynck2, Hugo Van hamme1, Patrick Wambacq1 1ESAT, KU Leuven, Belgium 2ELIS, Ghent University, Belgium {joris.pelemans,lyan.verwimp,hugo.vanhamme,patrick.wambacq}@esat.kuleuven.be kris.demuynck@elis.ugent.be Abstract In this paper we present SCALE, a new Python toolkit that contains two extensions to n-gram language models. The first extension is a novel technique to model compound words called Semantic Head Mapping (SHM). The second extension, Bag-of-Words Language Modeling (BagLM), bundles popular models such as Latent Semantic Analysis and Continuous Skip-grams. Both extensions scale to large data and allow the integration into first-pass ASR decoding. The toolkit is open source, includes working examples and can be found on http://github.com/jorispelemans/scale. Keywords: language engineering, compound modeling, bag of words 1. Introduction Although recurrent neural networks (RNNs) have recently become the state of the art in language modeling (Mikolov et al., 2010), they do not yet scale as well to large data as n-gram language models do and are as such impractical for automatic speech recognition (ASR). Even after opti- mization, an RNN language model (LM) is too computa- tionally expensive to be applied directly during decoding and its use is therefore restricted to a multi-pass approach in which the speech signal is first decoded using a simpler (often n-gram) LM, after which the output (N-best lists or lattices) is rescored using the RNNLM. In this paper we present SCALE, a new Python toolkit that contains two extensions to n-gram LMs: Semantic Head Mapping (SHM) and Bag-of-Words Language Modeling (BagLM). Both of them scale to large data and allow the in- tegration into first-pass ASR decoding. The toolkit is open source, includes working examples and can be found on http://github.com/jorispelemans/scale. 2. SHM for compound modeling One of the main issues in language modeling is data spar- sity: there is not enough training material to derive reliable statistics for every possible word sequence. This is in part caused by the ongoing formation of new words, a process that has only accelerated with the rise of the Internet e.g. Netizen, tweeple, infobesity. To successfully model these words, it is essential to look at lexicological processes and investigate the mechanisms that underly the formation of words. One of the most productive mechanisms in many languages is compounding, which induces the frequent creation of numerous new words all over the world. In this section we briefly describe SHM: a tool to find semantic heads for compounds. We also present some results that were achieved using this tool. For more details we refer the This research is funded by the Flemish government agency IWT (project 130041, SCATE) and by IWT–INNOVATIEF AAN- BESTEDEN and VRT in the STON project. reader to (Pelemans et al., 2014a) and (Pelemans et al., 2015). 2.1. Semantic Head Mapping In recent work (Pelemans et al., 2014a; Pelemans et al., 2015) we presented a novel clustering technique for com- pound words, called Semantic Head Mapping. We ar- gue that compounds are well represented by their seman- tic heads and as a consequence that compound-head clus- ters …

Content truncated to 500 words