Тест жиынтығы
Related Papers
Adapting Naturally Occurring Test Suites for Evaluation of Clinical Question Answering
This paper describes the structure of a test suite for evaluation of clinical question an- swering systems; presents several manually compiled resources found useful for test suite generation; and describes …...
RTG based surface realisation for TAG
Surface realisation with grammars inte- grating flat semantics is known to be NP complete. In this paper, we present a new algorithm for surface realisation based on Feature Based Tree …...
Test Suite Design for Biomedical Ontology Concept Recognition Systems
AbstractSystems that locate mentions of concepts from ontologies in free text are known as ontology concept recognition systems. This paper describes an approach to the evaluation of the workings of …...
Detecting Annotation Scheme Variation in Out-of-Domain Treebanks
AbstractTo ensure portability of NLP systems across multiple domains, existing treebanks are often extended by adding trees from interesting domains that were not part of the initial annotation effort. In …...
Discontinuous Verb Phrases in Parsing and Machine Translation of English and German
AbstractIn this paper, we focus on the verb-particle (V-Prt) split construction in English and German and its difficulty for parsing and Machine Translation (MT). For German, we use an existing …...
Fine-grained linguistic evaluation for state-of-the-art Machine Translation
AbstractThis paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 …...
The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation
AbstractDoes neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation. …...
A Linguistically Motivated Test Suite to Semi-Automatically Evaluate German–English Machine Translation Output
AbstractThis paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation …...
Discontinuous Constituency and BERT: A Case Study of Dutch
AbstractIn this paper, we set out to quantify the syntactic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite …...
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
AbstractNegation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing …...