Fluency

AbstractIn an effort to reduce the subjectivity, cost, and complexity of evaluation methods for machine translation (MT) and other language technologies, task-based assessment is examined as an alternative to metrics-based …...

In this paper, we present ParaEval, an automatic evaluation framework that uses paraphrases to improve the quality of machine translation evaluations. Previous work has focused on fixed n-gram evaluation metrics …...

In evaluating the output of language tech- nology applications—MT, natural language generation, summarisation—automatic eval- uation techniques generally conflate mea- surement of faithfulness to source content with fluency of the resulting …...

We present PEM, the first fully automatic met- ric to evaluate the quality of paraphrases, and consequently, that of paraphrase generation systems. Our metric is based on three crite- ria: …...

: Consecutive interpreting (CI) has gained increasing popularity and application in today’s world. Fluency is a criterion for CI output assessment. Although disfluencies among CI professionals were investigated in previous …...

Most previous research on automated speech scoring has focused on restricted, predictable speech. For automated scoring of unrestricted spontaneous speech, speech proficiency has been evaluated primarily on aspects of pro- …...

This paper presents a novel approach to pivot-based machine translation (MT): while the state-of-the-art uses two statisti- cal systems, this proposal treats the second system as a black box. Our …...

Generation for Focused Meeting Summarization Lu Wang Department of Computer Science Cornell University Ithaca, NY 14853 luwang@cs.cornell.edu Claire Cardie Department of Computer Science Cornell University Ithaca, NY 14853 cardie@cs.cornell.edu Abstract …...

We present a novel model, Freestyle, that learns to improvise rhyming and fluent re- sponses upon being challenged with a line of hip hop lyrics, by combining both bottom- up …...

A learner corpus is a useful resource for developing automatic assessment techniques for implementation in a computer-assisted language learning system. However, presently, learner corpora are only helpful in terms of …...

AbstractIn this paper we introduce MoBiL, a hybrid Monolingual, Bilingual and Language modelling feature set and feature selection and evaluation framework. The set includes translation quality indicators that can be …...

AbstractWe aim to shed light on the strengths and weaknesses of the newly introduced neural machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare …...

AbstractWe study automatic question generation for sentences from text passages in reading comprehension. We introduce an attention-based sequence learning model for the task and investigate the effect of encoding sentence- …...

AbstractThis paper presents a novel encoder-decoder model for automatically generating market comments from stock prices. The model first encodes both short- and long-term series of stock prices so that it …...

AbstractIn this work, we investigate the possibility of using fully automatic text simplification system on the English source in machine translation (MT) for improving its translation into an under-resourced language. …...

AbstractThe recent advances introduced by neural machine translation (NMT) are rapidly expanding the application fields of machine translation, as well as reshaping the quality level to be targeted. In particular, …...

AbstractUnderstanding contrastive opinions is a key component of argument generation. Central to an argument is the claim, a statement that is in dispute. Generating a counter-argument then requires generating a …...

AbstractMulti-turn conversations consist of complex semantic structures, and it is still a challenge to generate coherent and diverse responses given previous utterances. It’s practical that a conversation takes place under …...

AbstractRecent advances in pre-trained multilingual language models lead to state-of-the-art results on the task of quality estimation (QE) for machine translation. A carefully engineered ensemble of such models won the …...

AbstractWe propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM …...

AbstractNLP models are shown to suffer from robustness issues, i.e., a model’s prediction can be easily changed under small perturbations to the input. In this work, we present a Controlled …...

AbstractRecent question generation (QG) approaches often utilize the sequence-to-sequence framework (Seq2Seq) to optimize the log likelihood of ground-truth questions using teacher forcing. However, this training objective is inconsistent with actual …...

AbstractWe propose a novel text editing task, referred to as fact-based text editing, in which the goal is to revise a given document to better describe the facts in a …...

This paper discusses the tone acquisition of Chinese by students in the context of Indo- European languages. This paper conducted two experiments: (1) Using Zhang (2006)'s ‘Somatically-Enhanced Approach’(SEA) to conduct …...

AbstractRecent progress in pre-trained language models led to systems that are able to generate text of an increasingly high quality. While several works have investigated the fluency and grammatical correctness …...

AbstractExisting models for data-to-text tasks generate fluent but sometimes incorrect sentences e.g., “Nikkei gains” is generated when “Nikkei drops” is expected. We investigate models trained on contrastive examples i.e., incorrect …...

AbstractNeural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with limited emphasis …...

AbstractExisting pre-trained large language models have shown unparalleled generative capabilities. However, they are not controllable. In this paper, we propose MEGATRON-CNTRL, a novel framework that uses large-scale language models and …...

AbstractBack-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for …...

AbstractBuilding on recent advances in semantic parsing and text simplification, we investigate the use of semantic splitting of the source sentence as preprocessing for machine translation. We experiment with a …...

AbstractFor decades, chitchat bots are designed as a listener to passively answer what people ask. This passive and relatively simple dialogue mechanism gains less attention from humans and consumes the …...

AbstractTextual style transfer involves modifying the style of a text while preserving its content. This assumes that it is possible to separate style from content. This paper investigates whether this …...

AbstractMachine translation has an undesirable propensity to produce “translationese” artifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese …...

AbstractThis work describes an analysis of inter-annotator disagreements in human evaluation of machine translation output. The errors in the analysed texts were marked by multiple annotators under guidance of different …...

Non-native speakers of English tend to use in- appropriate expressions when relating to aca- demic writing. This may cause a non-fluent essay and probably unable to convey the pre- cise …...

AbstractAdversarial examples expose the vulnerabilities of natural language processing (NLP) models, and can be used to evaluate and improve their robustness. Existing techniques of generating such examples are typically driven …...

AbstractText style transfer aims to alter the style (e.g., sentiment) of a sentence while preserving its content. A common approach is to map a given sentence to content representation that …...

AbstractFraming involves the positive or negative presentation of an argument or issue depending on the audience and goal of the speaker. Differences in lexical framing, the focus of our work, …...

AbstractGenerating some appealing questions in open-domain conversations is an effective way to improve human-machine interactions and lead the topic to a broader or deeper direction. To avoid dull or deviated …...

AbstractThis work describes analysis of nature and causes of MT errors observed by different evaluators under guidance of different quality criteria: adequacy and comprehension and and a not specified generic …...

Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self- contradictory, and often lacks commonsense. Our analyses …...

We introduce a novel dataset of human judg- ments of machine-edited text and initial mod- els of those perceptions. Six machine-editing methods ranging from character swapping to variational autoencoders are …...

AbstractNatural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG …...

AbstractEvaluating video captioning systems is a challenging task as there are multiple factors to consider; for instance: the fluency of the caption, multiple actions happening in a single scene, and …...

AbstractDialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be …...

AbstractIn recent years, researchers have developed question-answering based approaches to automatically evaluate system summaries, reporting improved validity compared to word overlap-based metrics like ROUGE, in terms of correlation with human …...

AbstractGenerating factual, long-form text such as Wikipedia articles raises three key challenges: how to gather relevant evidence, how to structure information into well-formed text, and how to ensure that the …...

AbstractWhile often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research. We revisit this problem with a focus on producing consistent evaluations …...

AbstractRecent work on controlled text generation has either required attribute-based fine-tuning of the base language model (LM), or has restricted the parameterization of the attribute discriminator to be compatible with …...

AbstractWe present results from a study investigating how users perceive text quality and readability in extractive and abstractive summaries. We trained two summarisation models on Swedish news data and used …...

AbstractZero-shot paraphrase generation has drawn much attention as the large-scale high-quality paraphrase corpus is limited. Back-translation, also known as the pivot-based method, is typical to this end. Several works leverage …...

AbstractSome applications of artificial intelligence make it desirable that logical formulae be converted computationally to comprehensible natural language sentences. As there are many logical equivalents to a given formula, finding …...

AbstractIn this paper, we conduct the first study on spurious correlations for open-domain response generation models based on a corpus CGDialog curated by ourselves. The current models indeed suffer from …...