Amazon Mechanical Turk

We explore how to improve machine trans- lation systems by adding more translation data in situations where we already have substantial resources. The main challenge is how to buck the …...

AbstractThe collection and transcription of speech data is typically an expensive and time-consuming task. Voice over IP and cloud computing are poised to greatly reduce this impediment to research on …...

AbstractCreating more fine-grained annotated data than previously relevent document sets is important for evaluating individual components in automatic question answering systems. In this paper, we describe using the Amazon's Mechanical …...

Sentence fusion enables summarization and question-answering systems to produce out- put by combining fully formed phrases from different sentences. Yet there is little data that can be used to develop …...

The past few years have seen an increasing interest in using Amazon’s Mechanical Turk for purposes of collecting data and perform- ing annotation tasks. One such task is the mass …...

We investigate differences in point of view (POV) between two objective documents, where one is describing the subject matter in a more positive/negative way than the other, and present an …...

This paper investigates the feasibility of using crowd-sourcing services for the hu- man assessment of machine translation quality of translations into non-English tar- get languages. Non-expert graders are hired through …...

Semantic Textual Similarity (STS) measures the degree of semantic equivalence between two texts. This paper presents the results of the STS pilot task in Semeval. The training data contained 2000 …...

In this paper, we describe SemEval-2013 Task 4: the definition, the data, the evaluation and the results. The task is to capture some of the meaning of English noun compounds …...

We propose the use of a game with a purpose (GWAP) to facilitate crowd-sourcing of phrase- equivalents, as an alternative to expert or paid crowd-sourcing. Doodling is an online multi- …...

AbstractWe present the NewSoMe (News and Social Media) Corpus, a set of subcorpora with annotations on opinion expressions across genres (news reports, blogs, product reviews and tweets) and covering multiple …...

Verbal comprehension questions appear very frequently in Intelligence Quotient (IQ) tests, which measure human’s verbal ability includ- ing the understanding of the words with mul- tiple senses, the synonyms and …...

AbstractComputerized generation of humor is a notoriously difficult AI problem. We develop an algorithm called Libitum that helps humans generate humor in a Mad Lib, which is a popular fill-in-the-blank …...

AbstractWe present Speak, a toolkit that allows researchers to crowdsource speech audio recordings using Amazon Mechanical Turk (MTurk). Speak allows MTurk workers to submit speech recordings in response to a …...