Evaluation Method

This paper describes a method of evaluating quality for developers of machine translation sys- tems to easily check imperfections in their own systems. This evaluation method is a systematic, objective …...

. In the course of four evaluations in the Advanced Research Projects Agency Machine Translation series, evaluation methods have evolved for measuring the core components of a diverse set of …...

This paper describes a method of evaluating quality for developers of machine translation systems to easily check imperfections in their own systems. This evaluation method is a systematic, objective method …...

AbstractATR-MATRIX is a multi-lingual speech-to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel arrangement domain. In this paper, …...

An important first step in developing a cross-lingual question answering system is to understand whether techniques developed with English text will also work with other languages, such as Chinese. The …...

In order to promote the study of auto- matic summarization and translation, we need an accurate automatic evaluation method that is close to human evalua- tion. In this paper, we …...

In this paper a novel solution to auto- matic and unsupervised word sense induc- tion (WSI) is introduced. It represents an instantiation of the ‘one sense per colloca- tion’ observation …...

We introduce a novel evaluation scheme for the human evaluation of different machine translation systems. Our method is based on direct comparison of two sentences at a time by human …...

Previous work on statistical language gen- eration has primarily focused on grammat- icality and naturalness, scoring generation possibilities according to a language model or user feedback. More recent work has …...

We present a pilot study on an evaluation method which is able to rank translation out- puts with no reference translation, given only their source sentence. The system employs a …...

We present a novel evaluation method for grammatical error correction that addresses problems with previous approaches and scores systems in terms of improvement on the orig- inal text. Our method …...

AbstractGenerating keyphrases that summarize the main points of a document is a fundamental task in natural language processing. Although existing generative models are capable of predicting multiple keyphrases for an …...

Word embeddings are used in various fields of natural language processing. The use of word embeddings and concept or word sense em- beddings demonstrated effectiveness in many tasks, such as …...

AbstractTo assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or …...

AbstractWith the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic …...

AbstractThe Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the …...

AbstractStandard evaluations of Grammatical Error Correction (GEC) systems make use of a fixed reference text generated relative to the original text; they show, even when using multiple references, that we …...

AbstractMorality in dialogue systems has raised great attention in research recently. A moral dialogue system aligned with users’ values could enhance conversation engagement and user connections. In this paper, we …...