Test Set

Irregular (so-called broken) plural identification in modern standard Arabic is a problematic issue for information retrieval (IR) and language engineering applications, but their effect on the performance of IR has …...

If two translation systems differ differ in perfor- mance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we …...

Recognizing textual entailment is a chal- lenging problem and a fundamental com- ponent of many applications in natural language processing. We present a novel framework for recognizing textual entail- ment …...

The quality of a sentence translated by a machine translation (MT) system is dif- ficult to evaluate. We propose a method for automatically evaluating the quality of each translation. In …...

In this paper, we give a description of the ma- chine translation system developed at DCU that was used for our participation in the eval- uation campaign of the Third …...

Weights of the various components in a standard Statistical Machine Translation model are usually estimated via Minimum Error Rate Training. With this, one finds their optimum value on a development …...

We investigate two aspects of the empirical behavior of paired significance tests for NLP systems. First, when one system appears to outperform another, how does significance level relate in practice …...

In this paper, we investigate a challeng- ing task of automatic related work gener- ation. Given multiple reference papers as input, the task aims to generate a related work section …...

The efficacy of discriminative training in Statistical Machine Translation is heavily dependent on the quality of the develop- ment corpus used, and on its similarity to the test set. This …...

This paper describes LIMSI participation to the WMT’14 Shared Task on Qual- ity Estimation; we took part to the word- level quality estimation task for English to Spanish translations. Our …...

Named entity linking (NEL) can be applied to documents such as financial reports, web pages and news articles, but state of the art disambiguation techniques are currently too slow for …...

This paper presents a semantic parsing and reasoning approach to automatically solving math word problems. A new meaning representation language is de- signed to bridge natural language text and math …...

In this paper, we propose a new statistical method for sentiment analysis of figurative language within short texts collected from Twitter (called tweets) as a part of SemEval- 2015 Task …...

We use character-based statistical machine translation in order to correct user search queries in the e-commerce domain. The training data is automatically extracted from event logs where users re-issue their …...

Recent works have shown that the usage of a synthetic parallel corpus can be effec- tively exploited by a neural machine trans- lation system. In this paper, we propose a …...

Word sense disambiguation is necessary in translation because different word senses often have different translations. Neural machine translation models learn differ- ent senses of words as part of an end- …...

AbstractThis paper describes the Irony detection system that participates in SemEval-2018 Task 3: Irony detection in English tweets. The system participated in the subtasks A and B. This paper discusses …...

AbstractWe present a novel neural architecture for the Argument Reasoning Comprehension task of SemEval 2018. It is a simple neural network consisting of three parts, collectively judging whether the logic …...

AbstractThis paper describes our contribution to the SemEval-2019 Task 5 on the detection of hate speech against immigrants and women in Twitter (hatEval). We considered a supervised classification-based approach to …...

AbstractWe describe our submission to SemEval-2019 Task 4 on Hyperpartisan News Detection. We rely on a variety of engineered features originally used to detect propaganda. This is based on the …...

AbstractThis position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of …...

AbstractTask 12 of SemEval 2020 consisted of 3 subtasks, namely offensive language identification (Subtask A), categorization of offense type (Subtask B), and offense target identification (Subtask C). This paper presents …...

AbstractIn this paper, we present the system submitted to “SemEval-2020 Task 12”. The proposed system aims at automatically identify the Offensive Language in Arabic Tweets. A machine learning based approach …...

AbstractDespite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available …...

AbstractThis paper explains our teams’ submission to the Shared Task of Fine-Grained Propaganda Detection in which we propose a sequential BERT-CRF based Span Identification model where the fine-grained detection is …...

AbstractThis paper presents six document classification models using the latest transformer encoders and a high-performing ensemble model for a task of offensive language identification in social media. For the individual …...

AbstractThis paper describes our submission to the SemEval-2021 shared task on Lexical Complexity Prediction. We approached it as a regression problem and present an ensemble combining four systems, one feature-based …...

AbstractStandard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the …...

AbstractThis paper describes a method of semi-automatic word spotting in minority languages, from one and the same Aesop fable “The North Wind and the Sun” translated in Romance languages/dialects from …...

AbstractFor the task of classifying verbs in context as dynamic or stative, current models approach human performance, but only for particular data sets. To better understand the performance of such …...

AbstractThe quality of Natural Language Processing (NLP) models is typically measured by the accuracy or error rate of a predefined test set. Because the evaluation and optimization of these measures …...

AbstractThis paper introduces the related work and the results of Team Sapphire’s system for SemEval-2022 Task 4: Patronizing and Condescending Language Detection. We only participated in subtask 1. The task …...

AbstractThis paper describes the systems submitted for Marathi to Hindi low-resource speech translation task. Our primary submission is based on an end-to-end direct speech translation system, whereas the contrastive one …...

AbstractThis paper describes the method we submitted as the Janko team in the SemEval-2023 Task 2,Multilingual Complex Named Entity Recognition (MultiCoNER 2). We only participated in the Chinese track. In …...

AbstractThis paper describes our system for SemEval-2023 Task 3 Subtask 2 on Framing Detection. We used a multi-label contrastive loss for fine-tuning large pre-trained language models in a multi-lingual setting, …...

AbstractIn this paper, we introduce our submission in the task of visual word sense disambiguation (vWSD). Our proposed solution operates by deriving quasi-symbolic semantic categories from the hidden representations of …...

AbstractThe paper describes a transformer-based system designed for SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. The purpose of the task was to predict the intimacy of tweets in a range …...