Evaluation

I argue that because of spelling and typing errors and other properties of typed text, the identification of words and word boundaries in general requires syntactic and semantic knowledge. A …...

AbstractMachine-readable versions of everyday dictionaries have been seen as a likely source of information for use in natural language processing because they contain an enormous amount of lexical and semantic …...

In this paper, the Message Understanding Conferences are reviewed, and the natural language system evaluation that is underway in preparation for the next conference is described. The role of the …...

AbstractOne of the most important issues in the field of machine translation is evaluation of the translated sentences. This paper proposes a quantitative method of evaluation for machine translation systems. …...

This paper describes a system that provides customer service by allowing users to retrieve identification numbers of parts for medical systems using spoken natural language dialogue. The paper also presents …...

We describe a three-tiered approach for evaluation of spoken dialogue systems. The three tiers measure user satisfaction, system support of mission success and component performance. We describe our use of …...

This paper provides a method for generat- ing compact and efficient code to imple- ment the enforcement of a description in typed feature logic. It does so by view- ing …...

We present some preliminary results of a Czech-English translation system based on dependency trees. The fully auto- mated process includes: morphological tagging, analytical and tectogrammat- ical parsing of Czech, tectogrammati- …...

To be unambiguous about a Chinese geographic name represented in English text as Pinyin, one needs to recover the name in Chinese characters. We present our approach to this back-transliteration …...

We address the problem of sentence align- ment for monolingual corpora, a phe- nomenon distinct from alignment in par- allel corpora. Aligning large compara- ble corpora automatically would provide a …...

This system is an experiment of examples based approach. It is based on a corpus containing 220 thousand sentence pairs with word alignment. The system contains four parts: matching and …...

s Chun Xiao and Dietmar R¨osner Institut f¨ur Wissens- und Sprachverarbeitung Otto-von-Guericke Universit¨at Magdeburg Universit¨atsplatz 2, Magdeburg, Germany 39106 xiao|roesner@iws.cs.uni-magdeburg.de Abstract In this paper, we investigate the multiword verbs in …...

In this paper we present KNOWA, an English/Italian word aligner, developed at ITC-irst, which relies mostly on information contained in bilingual dictionaries. The performances of KNOWA are compared with those …...

. How can recent advances in automating the evaluation of machine translation (MT) engines be applied to automate the evaluation of more complex embedded MT systems? In this paper, we …...

AbstractThe research context of this paper is developing hybrid machine translation (MT) systems that exploit the advantages of linguistic rule-based and statistical MT systems. Arabic, as a morphologically rich language, …...

We present a novel approach for the automatic translation of written text into sign language. A new corpus focussing on the weather report domain for the language pair German and …...

In the paper we describe a dependency parser that uses exact search and global learning (Crammer et al., 2006) to produce labelled dependency trees. Our system inte- grates the task …...

In this paper we describe the MELB-MKB system, as entered in the SemEval-2007 lex- ical substitution task. The core of our sys- tem was the “Relatives in Context” unsuper- vised …...

. External methods for evaluating MT systems define various measures based on MT results and their usage. While operational systems are mostly evaluated since long by task- based methods, evaluation …...

This paper proposes the use of a language representation that specifies the relationship between terms of a sentence using question words. The proposed representation is tai- lored to help the …...

. In this paper, we first propose a terminology to be used in the French presentation of evaluation results of automatic summaries. In the second part of the paper, we …...

. In this article we present machine translationness (MTness, henceforth) as an approach for MT evaluations, in contrast to the notion of human likeness which is the basic criteria for …...

Question answering research has only recently started to spread from short factoid questions to more complex ones. One significant chal- lenge is the evaluation: manual evaluation is a difficult, time-consuming …...

. The strict character of most of the existing Machine Translation (MT) evaluation metrics does not permit them to capture lexical variation in translation. However, a central issue in MT …...

In contrast with other NLP tasks, only few and limited evaluation challenges have been carried out for terminology acquisition. It is nevertheless important to assess the progress made, the qual- …...

. In contrast with other NLP subtasks, there has been only few and limited evaluation campaigns for terminology acquisition. It is nevertheless important to assess the progress made, the quality …...

Topic segmentation was addressed by a large amount of work from which it is not easy to draw conclusions, especially about the need for knowl- edge. In this article, we …...

In this paper, we explore ways of improv- ing an inference rule collection and its ap- plication to the task of recognizing textual entailment. For this purpose, we start with …...

An important question in the evaluation of Natu- ral Language Generation systems concerns the re- lationship between textual characteristics and task performance. If the results of task-based evalua- tion can …...

The automatic analysis and classification of text using fine-grained attitude labels is the main task we address in our research. The de- veloped @AM system relies on compositio- nality principle …...

AbstractEvaluation of complex, collaborative dialogue systems is a difficult task. Traditionally, developers have relied upon subjective feedback from the user, and parametrisation over observable metrics. However, both models place some …...

AbstractThe focus of information retrieval evaluations, such as NIST’s TREC evaluations (e.g. Voorhees 2003), is on evaluation of the information content of system responses. On the other hand, retrieval tasks …...

AbstractLexical resources are basic components of many text processing system devoted to information extraction, question answering or dialogue. In paste years many resources have been developed such as FrameNet and …...

AbstractThis paper describes a hybrid system (FrAG) for tagging / parsing French text, and presents results from ongoing development work, corpus annotation and evaluation. The core of the system is …...

We present a novel fully unsupervised al- gorithm for POS induction from plain text, motivated by the cognitive notion of proto- types. The algorithm first identifies land- mark clusters of …...

We describe a novel approach to unsuper- vised learning of the events that make up a script, along with constraints on their temporal ordering. We collect natural- language descriptions of …...

AbstractOur purpose is to propose and discuss the latest version of an integrated method for dialogue analysis, annotation and evaluation., using a set of different pragmatic parameters. The annotation scheme …...

AbstractSign Languages (SLs) are the visuo-gestural languages practised by the deaf communities. Research on SLs requires to build, to analyse and to use corpora. The aim of this paper is …...

AbstractLanguage resources, including corpus and tools, are normally required to be combined in order to achieve a user’s specific task. However, resources tend to be developed independently in different, incompatible …...

AbstractThis paper describes a multi-lingual large-scale concept network obtained automatically by mining for concepts and relations and exploiting a variety of sources of knowledge from Wikipedia. Concepts and their lexicalizations …...

This paper addresses issues related to generat- ing feedback messages to errors related to Ar- abic verbs made by second language learners (SLLs). The proposed approach allows for in- dividualization. …...

AbstractThe usefulness of annotated corpora is greatly increased if there is an associated tool that can allow various kinds of operations to be performed in a simple way. Different kinds …...

As interest grows in the use of linguistically annotated corpora in research and teaching of foreign languages and literature, treebanks of various historical texts have been developed. We introduce the …...

This paper presents the results of an ex- perimental pilot user study, focusing on the evaluation of machine-translated us- er-generated content by users of an on- line community forum and …...

Live chats have become a popular form of communication, connecting people all over the globe. We believe that one of the simplest approaches for providing topic information to users joining …...

AbstractIn this paper we describe the research that was carried out and the resources that were developed within the DISCO (Development and Integration of Speech technology into COurseware for language …...

We present a new bilingual FrameNet lex- icon for English and German. It is cre- ated through a simple, but powerful ap- proach to construct a FrameNet in any language …...

Procedural dialog systems can help users achieve a wide range of goals. However, such systems are challenging to build, currently requiring manual engineering of substantial domain-specific task knowl- edge and …...

The syntactic ambiguity of a transitive verb (Vt) followed by a noun (N) has long been a problem in Chinese parsing. In this paper, we propose a classifier to resolve …...

AbstractWeb pages do not offer reliable metadata concerning their creation date and time. However, getting the document creation time is a necessary step for allowing to apply temporal normalization systems …...

Multi-document summarization (MDS) systems have been designed for short, un- structured summaries of 10-15 documents, and are inadequate for larger document collections. We propose a new approach to scaling up …...

AbstractThis article describes a large-scale evaluation of the use of Statistical Machine Translation for professional subtitling. The work was carried out within the FP7 EU-funded project SUMAT and involved two …...

Shell nouns, such as fact and problem, oc- cur frequently in all kinds of texts. These nouns themselves are unspecific, and can only be interpreted together with the shell content. …...

. Bilingual lexicons play a vital role in cross-language information retrieval and machine translation. The manual construction of these lexicons is often costly and time consuming. Word alignment techniques are …...

The terms “belief” and “factuality” both re- fer to the intention of the writer to present the propositional content of an utterance as firmly believed by the writer, not firmly …...

AbstractScripts are standardized event sequences describing typical everyday activities, which play an important role in the computational modeling of cognitive abilities (in particular for natural language processing). We present a …...

AbstractThis paper presents a German corpus for Named Entity Linking (NEL) and Knowledge Base Population (KBP) tasks. We describe the annotation guideline, the annotation process, NIL clustering techniques and conversion …...

AbstractThis paper describes the development and evaluation of a chatbot platform designed for the teaching/learning of Irish. The chatbot uses synthetic voices developed for the dialects of Irish. Speech-enabled chatbot …...

AbstractWith the success of word embedding methods in various Natural Language Processing tasks, all the field of distributional semantics has experienced a renewed interest. Beside the famous word2vec, recent studies …...

AbstractIn this paper we present our work on the usage of lexical resources for the Machine Translation English and Malayalam. We describe a comparative performance between different Statistical Machine Translation …...

AbstractWe report on the implementation of a morphological analyzer for the Sahidic dialect of Coptic, a now extinct Afro-Asiatic language. The system is developed in the finite-state paradigm. The main …...

AbstractWe present NorGramBank, a treebank for Norwegian with highly detailed LFG analyses. It is one of many treebanks made available through the INESS treebanking infrastructure. NorGramBank was constructed as a …...

We present a system for taxonomy construc- tion that reached the first place in all sub- tasks of the SemEval 2016 challenge on Tax- onomy Extraction Evaluation. Our simple yet …...

AbstractWord embeddings have been successfully used in several natural language processing tasks (NLP) and speech processing. Different approaches have been introduced to calculate word embeddings through neural networks. In the …...

We present a detailed evaluation and analysis of neural sequence-to-sequence models for text simplification on two distinct datasets: Wikipedia and Newsela. We employ both human and automatic evaluation to investigate …...

The Scielo database is an important source of scientific information in Latin America, containing articles from several research domains. A striking characteristic of Scielo is that many of its full-text …...

WordNets – lexical databases in which groups of synonyms are arranged according to the semantic relationships between them – are crucial resources in semantically-focused natural language processing tasks, but are …...

AbstractAutomatic extraction of semantic relations from text can support finding relevant information from scientific publications. We describe our participation in Task 7 of SemEval-2018 for which we experimented with two …...

This paper presents an interactive writing pro- totype system, SmartWrite, which provides Chinese word usage in the form of grammar patterns, collocations, and examples to assist language learners in writing. …...

AbstractThis paper describes the UMDSub system that participated in Task 2 of SemEval-2018. We developed a system that predicts an emoji given the raw text in a English tweet. The …...

Machine learning models have been providing promising results in many fields including natural language processing. These models are, nevertheless, prone to adversarial examples. These are artificially constructed examples which evince …...

AbstractNegation scope detection is widely performed as a supervised learning task which relies upon negation labels at word level. This suffers from two key drawbacks: (1) such granular annotations are …...

AbstractIn this paper we discuss the named entity recognition task for Russian texts related to cybersecurity. First of all, we describe the problems that arise in course of labeling unstructured …...

AbstractRhetoric is a vital element in modern poetry, and plays an essential role in improving its aesthetics. However, to date, it has not been considered in research on automatic poetry …...

AbstractWe develop and investigate several cross-lingual alignment approaches for neural sentence embedding models, such as the supervised inference classifier, InferSent, and sequential encoder-decoder models. We evaluate three alignment frameworks applied …...

AbstractThe research presented here is based on the theoretical model of corpus lattices. We implemented this as an effective data structure, and developed an algorithm based on this structure to …...

AbstractMulti-sentence compression (MSC) aims to generate a grammatical but reduced compression from multiple input sentences while retaining their key information. Previous dominating approach for MSC is the extraction-based word graph …...

AbstractStory generation, namely, generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural …...

AbstractWe present CLIRMatrix, a massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval extracted automatically from Wikipedia. CLIRMatrix comprises (1) BI-139, a bilingual dataset of queries in …...

AbstractNeural text generation (data- or text-to-text) demonstrates remarkable performance when training data is abundant which for many applications is not the case. To collect a large corpus of parallel data, …...

AbstractAchieving true human-like ability to conduct a conversation remains an elusive goal for open-ended dialogue systems. We posit this is because extant approaches towards natural language generation (NLG) are typically …...

AbstractLexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics. Evaluation is …...

AbstractExisting language models excel at writing from scratch, but many real-world scenarios require rewriting an existing document to fit a set of constraints. Although sentence-level rewriting has been fairly well-studied, …...

AbstractThis article reports on an ongoing project aiming at automatization of pseudonymization of learner essays. The process includes three steps: identification of personal information in an unstructured text, labeling for …...

AbstractExisting dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging …...

AbstractThe analytical description of charts is an exciting and important research area with many applications in academia and industry. Yet, this challenging task has received limited attention from the computational …...

AbstractEvaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of …...

AbstractTask-oriented dialogue systems typically rely on large amounts of high-quality training data or require complex handcrafted rules. However, existing datasets are often limited in size con- sidering the complexity of …...

AbstractGrowing interests have been attracted in Conversational Recommender Systems (CRS), which explore user preference through conversational interactions in order to make appropriate recommendation. However, there is still a lack of …...

AbstractEmpathy is a complex cognitive ability based on the reasoning of others’ affective states. In order to better understand others and express stronger empathy in dialogues, we argue that two …...

AbstractWhile counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types …...

AbstractEvaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better …...

AbstractLanguage models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand the …...

AbstractText style transfer is an important task in natural language generation, which aims to control certain attributes in the generated text, such as politeness, emotion, humor, and many others. It …...

AbstractIdentifying the high level structure of texts provides important information when performing distant reading analysis. The structure of texts is not necessarily linear, as transitions, such as changes in the …...

AbstractRecent progress in NLP is driven by pretrained models leveraging massive datasets and has predominantly benefited the world’s political and economic superpowers. Technologically underserved languages are left behind because they …...

AbstractOur goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would …...

AbstractThis project aimed at extending the test sets of the MuST-C speech translation (ST) corpus with new reference translations. The new references were collected from professional post-editors working on the …...

AbstractSummarization of legal case judgement documents is a challenging problem in Legal NLP. However, not much analyses exist on how different families of summarization models (e.g., extractive vs. abstractive) perform …...

AbstractText-to-SQL has attracted attention from both the natural language processing and database communities because of its ability to convert the semantics in natural language into SQL queries and its practical …...

AbstractWe participated in the WMT 2022 Large-Scale Machine Translation Evaluation for the African Languages Shared Task. This work describes our approach, which is based on filtering the given noisy data …...

AbstractIn Switzerland, two thirds of the population speak Swiss German, a primarily spoken language with no standardised written form. It is widely used on Swiss TV, for example in news …...

AbstractHealth coaching helps patients identify and accomplish lifestyle-related goals, effectively improving the control of chronic diseases and mitigating mental health conditions. However, health coaching is cost-prohibitive due to its highly …...

AbstractPoetry generation, and creative language generation in general, usually suffers from the lack of large training data. In this paper, we present a novel framework to generate sonnets that does …...

AbstractLong-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form …...

AbstractOne of the major problems with text simplification is the lack of high-quality data. The sources of simplification datasets are limited to Wikipedia and Newsela, restricting further development of this …...

AbstractTongue twisters are meaningful sentences that are difficult to pronounce. The process of automatically generating tongue twisters is challenging since the generated utterance must satisfy two conditions at once: phonetic …...