Кросс-энтропия

We present an extensive empirical com- parison of several smoothing techniques in the domain of language modeling, includ- ing those described by Jelinek and Mer- cer (1980), Katz (1987), and …...

The idea of '~familial relationships" among lan- guages is well-established and accepted, al- though some controversies persist in a few specific instances. By painstakingly record- ing and identifying regularities and …...

We consider the problem of computing the Kullback-Leibler distance, also called the relative entropy, between a probabilistic context-free grammar and a probabilistic fi- nite automaton. We show that there is …...

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which pro- duce …...

We investigate the task of performance pre- diction for language models belonging to the exponential family. First, we attempt to em- pirically discover a formula for predicting test set cross-entropy …...

. In this paper, we investigate the language model (LM) adaptation issue for Statis- tical Machine Translation (SMT). In order to overcome the weight bias on the LM obtained from …...

We report a wide range of comparative experiments establishing for the first time contrastive foundations for a completely unsupervised approach to bilingual grammar induction that is cognitively oriented toward early …...

We present a novel scheme of language modeling for a spoken dialogue system by effectively filtering query sentences collected via a Web site of wisdom of crowds. Our goal is …...

We introduce submodular optimization to the problem of training data subset selection for statistical machine translation (SMT). By explicitly formulating data selection as a submodular program, we ob- tain fast …...

This paper reports on the task of Native Language Identification (NLI). We devel- oped a machine learning system to identify the native language of authors of English texts written by …...

Natural language generation (NLG) is a critical component of spoken dialogue and it has a significant impact both on usabil- ity and perceived quality. Most NLG sys- tems in common …...

We describe a data selection method for domain adaptation in machine translation, based on relative frequency ratios computed between in-domain and out-of-domain corpora. Our method is compared to a state-of-the-art …...

AbstractIn this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text. In particular, we propose to learn neural rewards …...

AbstractExisting models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level. In this paper, we …...

AbstractNarrow specialized comparable corpora are often small in size. This particularity makes it difficult to build efficient models to acquire translation equivalents, especially for less frequent and rare words. One …...

AbstractNeural machine translation (NMT) has proven to be facilitated by curriculum learning which presents examples in an easy-to-hard order at different training stages. The keys lie in the assessment of …...

AbstractCurrent sequence-to-sequence models are trained to minimize cross-entropy and use softmax to compute the locally normalized probabilities over target sequences. While this setup has led to strong results in a …...

AbstractKnowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge …...

AbstractAlthough transformers are remarkably effective for many tasks, there are some surprisingly easy-looking regular languages that they struggle with. Hahn shows that for languages where acceptance depends on a single …...

AbstractSarcasm has gained notoriety for being difficult to detect by machine learning systems due to its figurative nature. In this paper, Bidirectional Encoder Representations from Transformers (BERT) model has been …...

AbstractThis paper describes our system used in the SemEval-2023 Task 11 Learning With Disagreements (Le-Wi-Di). This is a subjective task since it deals with detecting hate speech, misogyny and offensive …...

AbstractSubjectivity and difference of opinion are key social phenomena, and it is crucial to take these into account in the annotation and detection process of derogatory textual content. In this …...