Towards Quantum Language Models

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1840–1849 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Towards Quantum Language Models Ivano Basile Scuola Normale Superiore, Pisa, Italy ivano.basile@sns.it Fabio Tamburini FICLIT - University of Bologna, Italy fabio.tamburini@unibo.it Abstract This paper presents a new approach for building Language Models using the Quantum Probability Theory, a Quantum Language Model (QLM). It mainly shows that relying on this probability calculus it is possible to build stochastic models able to benefit from quantum correlations due to interference and entanglement. We ex- tensively tested our approach showing its superior performances, both in terms of model perplexity and inserting it into an automatic speech recognition evaluation setting, when compared with state-of-the- art language modelling techniques. 1 Introduction Quantum Mechanics Theory (QMT) is one of the most successful theories in modern science. De- spite its effectiveness in the physics realm, the at- tempts to apply it in other domains remain quite limited, excluding, of course, the large quantity of studies regarding Quantum Information Process- ing on quantum computers. Only in recent years some scholars tried to em- body principles derived from QMT into their spe- cific fields, for example, by the Information Re- trieval community (Zuccon et al., 2009; Melucci and van Rijsbergen, 2011; Gonz´alez and Caicedo, 2011; Melucci, 2015) and in the domain of cog- nitive sciences and decision making (Khrennikov, 2010; Busemeyer and Bruza, 2012; Aerts et al., 2013). In the machine learning field (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2017) have used unitary evolution matrices for building deep neural networks obtaining interesting results, but we have to observe that their works do not ad- here to QMT and use unitary evolution operators in a way not allowed by QMT. In recent years, also the Natural Language Processing (NLP) commu- nity started to look at QMT with interest and some studies using it have already been presented (Bla- coe et al., 2013; Liu et al., 2013; Tamburini, 2014; Kartsaklis et al., 2016). Language models (LM) are basic tools in NLP used in various applications, such as Automatic Speech Recognition (ASR), machine translation, part-of-speech tagging, etc., and were traditionally modeled by using N-grams and various smoothing techniques. Among the dozen of tools for comput- ing N-gram LM, we will refer to CMU-SLM (with Good-Turing smoothing) (Clarkson and Rosen- feld, 1997) and IRSTLM (with Linear Witten-Bell smoothing) (Federico et al., 2008); the latter is the tool used in Kaldi (Povey et al., 2011b), one of the most powerful and used open-source ASR pack- age that we will use for some of the experiments presented in the following sections. In recent years new techniques from the Neural Networks (NN) domain have been introduced in order to enhance the performances of such models. Elman recurrent NN, as used in the RNNLM tool (Mikolov et al., 2010, 2011), or Long Short-Term Memory NN, as in the tool LSTMLM (Soutner and M¨uller, 2015), produce state-of-the-art perfor- mances for …

Content truncated to 500 words