Evaluation Protocol
Related Papers
Bilingual Lexicon Induction: Effortless Evaluation of Word Alignment Tools and Production of Resources for Improbable Language Pairs
AbstractIn this paper, we present a simple protocol to evaluate word aligners on bilingual lexicon induction tasks from parallel corpora. Rather than resorting to gold standards, it relies on a …...
Evaluation Protocol and Tools for Question-Answering on Speech Transcripts
AbstractQuestion Answering (QA) technology aims at providing relevant answers to natural language questions. Most Question Answering research has focused on mining document collections containing written texts to answer written questions. …...
Evaluating Web-as-corpus Topical Document Retrieval with an Index of the OpenDirectory
AbstractThis article introduces a novel protocol and resource to evaluate Web-as-corpus topical document retrieval. To the contrary of previous work, our goal is to provide an automatic, reproducible and robust …...
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation
AbstractRecent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. …...
Dysarthric speech evaluation: automatic and perceptual approaches
Perceptual evaluation is still the most common method in clinical practice for the diagnosis and monitoring of the condition progression of people suffering from dysarthria (or speech disorders more generally). …...
Revisiting the Task of Scoring Open IE Relations
Knowledge Base Completion infers missing facts from existing ones in knowledge bases. As recent Open Information Extraction systems allow us to extract ever larger (yet incomplete) open-domain Knowledge Bases from …...
Revisiting the Evaluation of Theory of Mind through Question Answering
AbstractTheory of mind, i.e., the ability to reason about intents and beliefs of agents is an important task in artificial intelligence and central to resolving ambiguous references in natural language …...
A Re-evaluation of Knowledge Graph Completion Methods
AbstractKnowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. A vast number of state-of-the-art KGC techniques have got published at top conferences in several research …...
Implicit Discourse Relation Classification: We Need to Talk about Evaluation
AbstractImplicit relation classification on Penn Discourse TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understanding of discourse relations. However, the lack of consistency in preprocessing and evaluation …...
Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future
AbstractAnnotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. …...
Rogue Scores
AbstractCorrect, comparable, and reproducible model evaluation is essential for progress in machine learning. Over twenty years, thousands of language and vision models have been evaluated with a popular metric called …...