Perplexity

A new natural language system, TINA, has been developed for applications involving speech understanding tasks, which integrates key ideas from context free grammars, Augmented Transition Networks (ATN's) [1], and Lexical …...

We describe two attempt to improve our stochastic language models. In the first, we identify a systematic overestimation in the traditional backoff model, and use statisticalreasoning to correct it. Our …...

° • We present an iterative procedure to build a Chinese language model (LM). We seg- ment Chinese text into words based on a word-based Chinese language model. How- ever, …...

We consider the use of language models whose size and accuracy are intermedi- ate between different order n-gram models. Two types of models are studied in partic- ular. Aggregate Markov …...

In this paper, we propose a method for con- structing bigram LR tables by way of incor- porating bigram constraints into an LR table. Using a bigram LR table, it …...

A statistical language model may be used to segment a data sequence by thresholding its instantaneous entropy. In this paper we de- scribe how this process works, and we apply …...

In this paper, we present a stochastic language mod- eling tool which aims at retrieving variable-length phrases (multigrams), assuming bigram dependen- cies between them. The phrase retrieval can be in- …...

Language models for speech recognition con- centrate solely on recognizing the words that were spoken. In this paper, we advocate re- defining the speech recognition problem so that its goal …...

We study the impact of richer syntac- tic dependencies on the performance of the structured language model (SLM) along three dimensions: parsing accu- racy (LP/LR), perplexity (PPL) and word- error-rate …...

A new almost-parsing language model incorporat- ing multiple knowledge sources that is based upon the concept of Constraint Dependency Grammars is presented in this paper. Lexical features and syn- tactic …...

We investigate the performance of the Structured Language Model (SLM) in terms of perplexity (PPL) when its compo- nents are modeled by connectionist mod- els. The connectionist models use a …...

In this paper, we explore the use of Random Forests (RFs) (Amit and Geman, 1997; Breiman, 2001) in language modeling, the problem of predicting the next word based on words …...

Theoretically, an improvement in a language model occurs as the size of the n-grams increases from 3 to 5 or higher. As the n-gram size increases, the number of parameters …...

Previous work shows that the process of parallel text exploitation to extract transfer mappings between language pairs raises the capability of language translation. However, while this process can be fully …...

The conventional n-gram language model exploits only the immediate context of historical words without exploring long-distance semantic information. In this paper, we present a new information source extracted from latent …...

In this paper, we explore statistical lan- guage modelling for a speech-enabled MP3 player application by generating a corpus from the interpretation grammar written for the application with the Gram- …...

Adapting language models across styles and topics, such as for lecture transcrip- tion, involves combining generic style models with topic-specific content rele- vant to the target document. In this work, …...

Grounded language models represent the rela- tionship between words and the non-linguistic context in which they are said. This paper de- scribes how they are learned from large cor- pora …...

The recent availability of large corpora for training N-gram language models has shown the utility of models of higher or- der than just trigrams. In this paper, we investigate methods …...

Modern unsupervised POS taggers usually apply an optimization procedure to a non- convex function, and tend to converge to local maxima that are sensitive to start- ing conditions. The quality …...

AbstractThis paper investigates the effectiveness of online temporal language model adaptation when applied to a Thai broadcast news transcription task. Our adaptation scheme works as follow: first an initial language …...

Using multi-layer neural networks to esti- mate the probabilities of word sequences is a promising research area in statistical lan- guage modeling, with applications in speech recognition and statistical machine …...

This paper presents an attempt at building a large scale distributed composite language model that simultaneously accounts for local word lexical information, mid-range sentence syntactic structure, and long-span document semantic …...

Building on earlier work that integrates dif- ferent factors in language modeling, we view (i) backing off to a shorter history and (ii) class-based generalization as two complemen- tary mechanisms …...

Augmented and alternative communication (AAC) devices enable users with certain com- munication disabilities to participate in every- day conversations. Such devices often rely on statistical language models to improve text …...

We investigate a language model that com- bines morphological and shape features with a Kneser-Ney model and test it in a large crosslingual study of European languages. Even though the …...

Among various neural network language models (NNLMs), recurrent neural network-based lan- guage models (RNNLMs) are very competitive in many cases. Most current RNNLMs only use one single feature stream, i.e., …...

We present a novel scheme of language modeling for a spoken dialogue system by effectively filtering query sentences collected via a Web site of wisdom of crowds. Our goal is …...

AbstractIn this paper we deal with named entity detection on data acquired via OCR process on documents dating from 1890. The resulting corpus is very noisy. We perform an analysis …...

In this paper we examine language mod- eling for text simplification. Unlike some text-to-text translation tasks, text simplifi- cation is a monolingual translation task al- lowing for text in both …...

We introduce a class of probabilistic con- tinuous translation models called Recur- rent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do …...

We introduce a novel approach for build- ing language models based on a system- atic, recursive exploration of skip n-gram models which are interpolated using modi- fied Kneser-Ney smoothing. Our …...

We propose the use of a generative model to simulate user behaviour in a novel task- oriented dialog domain, where user goals are spatial routes across artificial land- scapes. We …...

. This work is motivated by the will of creating a new part-of-speech annotated corpus in French from an existing one. We propose a general and operational definition of the …...

We propose a language modeling (LM) approach incorporating interpolated distanced n-grams in a Dirichlet class language model (DCLM) (Chien and Chueh, 2011) for speech recognition. The DCLM relaxes the bag-of-words …...

We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of …...

This paper studies the idea of remov- ing low-frequency words from a corpus, which is a common practice to reduce computational costs, from a theoretical standpoint. Based on the assumption …...

We propose the first implementation of an infinite-order generative dependency model. The model is based on a new recursive neural network architecture, the Inside-Outside Recursive Neural Network. This architecture allows …...

Post-editing the output of a statistical machine translation (SMT) system to obtain high-quality translation has become an increasingly common application of SMT, which henceforth we re- fer to as post-editing-based …...

AbstractThe role of language models in SMT is to promote fluent translation output, but traditional n-gram language models are unable to capture fluency phenomena between distant words, such as some …...

In the last several years, neural network models have significantly improved accu- racy in a number of NLP tasks. How- ever, one serious drawback that has im- peded their adoption …...

We present persona-based models for han- dling the issue of speaker consistency in neural response generation. A speaker model encodes personas in distributed em- beddings that capture individual charac- teristics …...

AbstractThis paper describes a Bayesian language model for predicting spontaneous utterances. People sometimes say unexpected words, such as fillers or hesitations, that cause the miss-prediction of words in normal N-gram …...

Part of the unique cultural heritage of Chi- na is the Chinese couplet. Given a sen- tence (namely an antecedent clause), peo- ple reply with another sentence (namely a subsequent …...

Neural networks are among the state-of- the-art techniques for language modeling. Existing neural language models typically map discrete words to distributed, dense vector representations. After information processing of the preceding …...

AbstractThis work presents a straightforward method for extending or creating in-domain web corpora by focused webcrawling. The focused webcrawler uses statistical N-gram language models to estimate the relatedness of documents …...

AbstractNeural network training has been shown to be advantageous in many natural language processing applications, such as language modelling or machine translation. In this paper, we describe in detail a …...

We present a probabilistic model of the in- fluence of language on the behavior of the U.S. Supreme Court, specifically influence of amicus briefs on Court decisions and opin- ions. …...

In this work, we propose a novel method to incorporate corpus-level discourse infor- mation into language modelling. We call this larger-context language model. We in- troduce a late fusion approach …...

In this paper we introduce Latent Tree Lan- guage Model (LTLM), a novel approach to language modeling that encodes syntax and semantics of a given sentence as a tree of …...

Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic in- formation. In this paper we show that the strong learning capability of …...

Semantic error detection and correction is an important task for applications such as fact checking, speech-to-text or grammatical er- ror correction. Current approaches gener- ally focus on relatively shallow semantics …...

AbstractPredictive incremental parsing produces syntactic representations of sentences as they are produced, e.g. by typing or speaking. In order to generate connected parses for such unfinished sentences, upcoming word types …...

In this work we present a generalisation of the Modified Kneser-Ney interpolative smoothing for richer smoothing via additional discount parameters. We provide mathematical under- pinning for the estimator of the …...

AbstractTraining good word embeddings requires large amounts of data. Out-of-vocabulary words will still be encountered at test-time, leaving these words without embeddings. To overcome this lack of embeddings for rare …...

AbstractWe release Galactic Dependencies 1.0—a large set of synthetic languages not found on Earth, but annotated in Universal Dependencies format. This new resource aims to provide training and development data …...

Natural language understanding often re- quires deep semantic knowledge. Ex- panding on previous proposals, we suggest that some important aspects of semantic knowledge can be modeled as a language model …...

AbstractHuman verbal communication includes affective messages which are conveyed through use of emotionally colored words. There has been a lot of research effort in this direction but the problem of …...

AbstractWe present a Character-Word Long Short-Term Memory Language Model which both reduces the perplexity with respect to a baseline word-level language model and reduces the number of parameters of the …...

We describe a data selection method for domain adaptation in machine translation, based on relative frequency ratios computed between in-domain and out-of-domain corpora. Our method is compared to a state-of-the-art …...

AbstractPOS tagging and dependency parsing achieve good results for homogeneous datasets. However, these tasks are much more difficult on heterogeneous datasets. In (Mukherjee et al. 2016, 2017), we address this …...

AbstractLanguage models are typically applied at the sentence level, without access to the broader document context. We present a neural language model that incorporates document context in the form of …...

AbstractMany natural and formal languages contain words or symbols that require a matching counterpart for making an expression well-formed. The combination of opening and closing brackets is a typical example …...

AbstractWe propose a new generative language model for sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional …...

AbstractEmbedding and projection matrices are commonly used in neural language models (NLM) as well as in other sequence processing networks that operate on large vocabularies. We examine such matrices in …...

AbstractDistributed representations of words play a major role in the field of natural language processing by encoding semantic and syntactic information of words. However, most existing works on learning word …...

AbstractA context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is …...

AbstractSelf-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. In the context of language modeling, this property is particularly appealing as it …...

AbstractRNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here …...

AbstractLanguage models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., …...

AbstractNeural language models are usually trained using Maximum-Likelihood Estimation (MLE). The corresponding objective function for MLE is derived from the Kullback-Leibler (KL) divergence between the empirical probability distribution representing the …...

AbstractHighly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the …...

AbstractSpectral models for learning weighted non-deterministic automata have nice theoretical and algorithmic properties. Despite this, it has been challenging to obtain competitive results in language modeling tasks, for two main …...

AbstractTraditional language models are unable to efficiently model entity names observed in text. All but the most popular named entities appear infrequently in text providing insufficient context. Recent efforts have …...

AbstractIn recent years, advances in neural variational inference have achieved many successes in text processing. Examples include neural topic models which are typically built upon variational autoencoder (VAE) with an …...

AbstractIn recent years there has been a burgeoning interest in the use of computational methods to distinguish between elicited speech samples produced by patients with dementia, and those from healthy …...

AbstractIn the paper, we present class-based LSTM Russian language models (LMs) with classes generated with the use of both word frequency and linguistic information data, obtained with the help of …...

AbstractIn this paper, we introduce personalized word embeddings, and examine their value for language modeling. We compare the performance of our proposed prediction model when using personalized versus generic word …...

AbstractMultilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We generate randomly ordered transformers and train them …...

AbstractNon-goal oriented dialog agents (i.e. chatbots) aim to produce varying and engaging conversations with a user; however, they typically exhibit either inconsistent personality across conversations or the average personality of …...

AbstractLanguage modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which …...

AbstractPast work on story generation has demonstrated the usefulness of conditioning on a generation plan to generate coherent stories. However, these approaches have used heuristics or off-the-shelf models to first …...

AbstractIn order to combat overfitting and in pursuit of better generalization, label smoothing is widely applied in modern neural machine translation systems. The core idea is to penalize over-confident outputs …...

AbstractThe uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanation for …...

AbstractTransformers have advanced the field of natural language processing (NLP) on a variety of important tasks. At the cornerstone of the Transformer architecture is the multi-head attention (MHA) mechanism which …...

AbstractManually annotating a treebank is time-consuming and labor-intensive. We conduct delexicalized cross-lingual dependency parsing experiments, where we train the parser on one language and test on our target language. As …...

AbstractSelf-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic computation and memory requirements with respect to sequence length. Successful …...

AbstractTransformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur …...

AbstractLanguage model fine-tuning is essential for modern natural language processing, but is computationally expensive and time-consuming. Further, the effectiveness of fine-tuning is limited by the inclusion of training examples that …...

AbstractReliable automatic evaluation of dialogue systems under an interactive environment has long been overdue. An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve …...

AbstractWe analyse how a transformer-based language model learns the rules of chess from text data of recorded games. We show how it is possible to investigate how the model capacity …...

AbstractTransformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of …...

AbstractBuilding pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance? We propose an alternative to larger training sets by automatically identifying …...

AbstractClass-based language models (LMs) have been long devised to address context sparsity in n-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize …...

AbstractCurrent language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. The standard language modeling setup fails to address these issues. In this …...

AbstractDeep learning (DL) techniques involving fine-tuning large numbers of model parameters have delivered impressive performance on the task of discriminating between language produced by cognitively healthy individuals, and those with …...

AbstractGender-neutral pronouns have recently been introduced in many languages to a) include non-binary people and b) as a generic singular. Recent results from psycholinguistics suggest that gender-neutral pronouns (in Swedish) …...

AbstractNatural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the …...

AbstractWe present a novel architecture for explainable modeling of task-oriented dialogues with discrete latent variables to represent dialogue actions. Our model is based on variational recurrent neural networks (VRNN) and …...

AbstractWe present GTP-SW3, a 3.5 billion parameter autoregressive language model, trained on a newly created 100 GB Swedish corpus. This paper provides insights with regards to data collection and training, …...

AbstractWe present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for …...

AbstractNeural language models (LMs) such as GPT-2 estimate the probability distribution over the next word by a softmax over the vocabulary. The softmax layer produces the distribution based on the …...

AbstractModel-based, reference-free evaluation metricshave been proposed as a fast and cost-effectiveapproach to evaluate Natural Language Generation(NLG) systems. Despite promising recentresults, we find evidence that reference-freeevaluation metrics of summarization and dialoggeneration …...

AbstractPre-trained multilingual models, such as mBERT, XLM-R and mT5, are used to improve the performance on various tasks in low-resource languages via cross-lingual transfer. In this framework, English is usually …...

AbstractRecent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at testing time or represent them using …...

AbstractWe introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which …...

AbstractMulti-Document Summarization (MDS) commonly employs the 2-stage extract-then-abstract paradigm, which first extracts a relatively short meta-document, then feeds it into the deep neural networks to generate an abstract. Previous work …...

AbstractAdversarial texts help explore vulnerabilities in language models, improve model robustness, and explain their working mechanisms. However, existing word-level attack methods trap in a one-to-one attack pattern, i.e., only a …...

AbstractWe present DiffusionBERT, a new generative masked language model based on discrete dif- fusion models. Diffusion models and many pre- trained language models have a shared training objective, i.e., denoising, …...

AbstractTopic modeling is a commonly used method for identifying and extracting topics from a corpus of documents. While several evaluation techniques, such as perplexity and topic coherence, have been developed …...

AbstractFederated learning is a training paradigm that learns from multiple distributed users without aggregating data on a centralized server, promising the ability to deploy machine-learning to a diverse population of …...

AbstractWhen acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities. Is this preference due to a learning bias for hierarchical structure, or due to more general biases that …...

AbstractToday’s probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics (e.g., perplexity). This …...

AbstractDirectly learning from complex examples is generally problematic for humans and machines. Indeed, a better strategy is exposing learners to examples in a reasonable, pedagogically-motivated order. Curriculum Learning (CL) has …...

AbstractIn this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly …...

AbstractWe introduce our probabilistic robustness rewarded data optimization (PRoDO) approach as a framework to enhance the model’s generalization power by selecting training data that optimizes our probabilistic robustness metrics. We …...

AbstractScaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different …...