Evaluation Methodology

This paper considers key elements of evaluation methodol- ogy, indicating the many points involved and advocating an unpacking approach in specifying an evaluation remit and de- sign. Recognising the importance …...

The Defense Advanced Research Projects Agency (DARPA) Machine Translation (MT) Initiative spanned four years. One outcome of this effort was a methodology for evaluating the core technology of MT systems …...

AbstractThis article describes the first CHIL evaluation campaign in which 12 technologies were evaluated. The major outcomes of the first evaluation campaign are the so-called Evaluation Packages. An evaluation package …...

Obtaining large volumes of inference knowl- edge, such as entailment rules, has become a major factor in achieving robust seman- tic processing. While there has been sub- stantial research on …...

This paper presents three successful tech- niques to translate prepositions heading verbal complements by means of rich lin- guistic information, in the context of a rule-based Machine Translation system for …...

We present a method that paraphrases a given sentence by first generating candidate para- phrases and then ranking (or classifying) them. The candidates are generated by ap- plying existing paraphrasing …...

We propose a new shared task on grading stu- dent answers with the goal of enabling well- targeted and flexible feedback in a tutorial di- alogue setting. We provide an …...

In this paper, we propose an unsupervised method to identify noun sense changes based on rigorous analysis of time-varying text data available in the form of millions of digitized books. …...

Most of previous work in knowledge base (KB) completion has focused on the problem of relation extraction. In this work, we focus on the task of inferring missing entity type …...

We provide an analysis of current evalua- tion methodologies applied to summariza- tion metrics and identify the following ar- eas of concern: (1) movement away from evaluation by correlation with …...

AbstractUnsupervised learning of morphological segmentation of words in a language, based only on a large corpus of words, is a challenging task. Evaluation of the learned segmentations is a challenge …...

AbstractCross document event coreference (CDEC) is an important task that aims at aggregating event-related information across multiple documents. We revisit the evaluation for CDEC, and discover that past works have …...

AbstractEmotion Representation Mapping (ERM) has the goal to convert existing emotion ratings from one representation format into another one, e.g., mapping Valence-Arousal-Dominance annotations for words or sentences into Ekman’s Basic …...

The quality of training data is one of the crucial problems when a learning-centered approach is employed. This paper proposes a new method to investigate the quality of a large …...

AbstractOnline texts - across genres, registers, domains, and styles - are riddled with human stereotypes, expressed in overt or subtle ways. Word embeddings, trained on these texts, perpetuate and amplify …...

AbstractCurrent intelligent systems need the expensive support of machine learning experts to sustain their performance level when used on a daily basis. To reduce this cost, i.e. remaining free from …...

AbstractMachine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we …...