Articles

This paper describes the LeBLEU evalu- ation score for machine translation, sub- mitted to WMT15 Metrics Shared Task. LeBLEU extends the popular BLEU score to consider fuzzy matches between word n-grams. While there are several variants of BLEU that allow to non-exact matches between words either by character-based distance measures …...

This paper describes USAAR’s submis- sion to the the metrics shared task of the Workshop on Statistical Machine Transla- tion (WMT) in 2015. The goal of our sub- mission is to take advantage of the seman- tic overlap between hypothesis and ref- erence translation for predicting MT out- put adequacy …...

Through using knowledge bases, ques- tion answering (QA) systems have come to be able to answer questions accurately over a variety of topics. However, knowl- edge bases are limited to only a few ma- jor languages, and thus it is often nec- essary to build QA systems that answer questions …...

In English-to-Japanese translation, BLEU (Papineni et al., 2002), the de facto standard evaluation metric for machine translation (MT), has very weak corre- lation with human judgments (Goto et al., 2011; Goto et al., 2013). Therefore, RIBES (Isozaki et al., 2010; Hirao et al., 2014) was proposed. RIBES mea- sures similarity …...

We show that, consistent with MEANT- tuned systems that translate into Chinese, MEANT-tuned MT systems that trans- late into English also outperforms BLEU- tuned systems across commonly used MT evaluation metrics, even in BLEU. The re- sult is achieved by significantly improv- ing MEANT’s sentence-level ranking cor- relation with human …...

We describe the submissions of ILLC UvA to the metrics and tuning tasks on WMT15. Both submissions are based on the BEER evaluation metric origi- nally presented on WMT14 (Stanojevi´c and Sima’an, 2014a). The main changes introduced this year are: (i) extending the learning-to-rank trained sentence level metric to the …...

We propose the use of character n-gram F-score for automatic evaluation of ma- chine translation output. Character n- grams have already been used as a part of more complex metrics, but their individual potential has not been investigated yet. We report system-level correlations with hu- man rankings for 6-gram F1-score …...

This paper describes Meteor-WSD and RATATOUILLE, the LIMSI submissions to the WMT15 metrics shared task. Meteor- WSD extends synonym mapping to lan- guages other than English based on align- ments and gives credit to semantically adequate translations in context. We show that context-sensitive synonym se- lection increases the correlation of …...

This paper presents our metric (UoW- LSTM) submitted in the WMT-15 met- rics task. Many state-of-the-art Machine Translation (MT) evaluation metrics are complex, involve extensive external re- sources (e.g. for paraphrasing) and require tuning to achieve the best results. We use a metric based on dense vector spaces and Long …...

AbstractThis paper formalizes the problem of solving multi-sentence algebraic word problems as that of generating and scoring equation trees. We use integer linear programming to generate equation trees and score their likelihood by learning local and global discriminative models. These models are trained on a small set of word problems …...

AbstractWe propose a new method for semantic parsing of ambiguous and ungrammatical input, such as search queries. We do so by building on an existing semantic parsing framework that uses synchronous context free grammars (SCFG) to jointly model the input sentence and output meaning representation. We generalize this SCFG framework …...

AbstractWe study the generalization of maximum spanning tree dependency parsing to maximum acyclic subgraphs. Because the underlying optimization problem is intractable even under an arc-factored model, we consider the restriction to noncrossing dependency graphs. Our main contribution is a cubic-time exact inference algorithm for this class. We extend this algorithm …...

AbstractSemantic parsers conventionally construct logical forms bottom-up in a fixed order, resulting in the generation of many extraneous partial logical forms. In this paper, we combine ideas from imitation learning and agenda-based parsing to train a semantic parser that searches partial logical forms in a more strategic order. Empirically, our …...

AbstractWe present DefIE, an approach to large-scale Information Extraction (IE) based on a syntactic-semantic analysis of textual definitions. Given a large corpus of definitions we leverage syntactic dependencies to reduce data sparsity, then disambiguate the arguments and content words of the relation strings, and finally exploit the resulting information to …...

AbstractWe present a novel hierarchical distance-dependent Bayesian model for event coreference resolution. While existing generative models for event coreference resolution are completely unsupervised, our model allows for the incorporation of pairwise distances between event mentions — information that is widely used in supervised coreference models to guide the generative clustering …...

AbstractWe show how to train the fast dependency parser of Smith and Eisner (2008) for improved accuracy. This parser can consider higher-order interactions among edges while retaining O(n3) runtime. It outputs the parse with maximum expected recall—but for speed, this expectation is taken under a posterior distribution that is constructed …...

AbstractWe present the first large-scale, corpus based verification of Dowty’s seminal theory of proto-roles. Our results demonstrate both the need for and the feasibility of a property-based annotation scheme of semantic relationships, as opposed to the currently dominant notion of categorical roles....

AbstractThe observed pronunciations or spellings of words are often explained as arising from the “underlying forms” of their morphemes. These forms are latent strings that linguists try to reconstruct by hand. We propose to reconstruct them automatically at scale, enabling generalization to new words. Given some surface word types of …...

AbstractWe present Plato, a probabilistic model for entity resolution that includes a novel approach for handling noisy or uninformative features, and supplements labeled training data derived from Wikipedia with a very large unlabeled text corpus. Training and inference in the proposed model can easily be distributed across many servers, allowing …...

AbstractMachine learning approaches to coreference resolution vary greatly in the modeling of the problem: while early approaches operated on the mention pair level, current research focuses on ranking architectures and antecedent trees. We propose a unified representation of different approaches to coreference resolution in terms of the structure they operate …...