WAGS: A Beautiful English-Italian Benchmark Supporting Word Alignment Evaluation on Rare Words

WAGS: A Beautiful English-Italian Benchmark Supporting Word Alignment Evaluation on Rare Words Luisa Bentivogli1, Mauro Cettolo1, M. Amin Farajian1,2, Marcello Federico1 1Fondazione Bruno Kessler, Via Sommarive 18, Povo - Trento, Italy 2University of Trento, Via Sommarive, 5, Povo - Trento, Italy {bentivo,cettolo,farajian,federico}@fbk.eu Abstract This paper presents WAGS (Word Alignment Gold Standard), a novel benchmark which allows extensive evaluation of WA tools on out-of-vocabulary (OOV) and rare words. WAGS is a subset of the Common Test section of the Europarl English-Italian parallel corpus, and is specifically tailored to OOV and rare words. WAGS is composed of 6,715 sentence pairs containing 11,958 occurrences of OOV and rare words up to frequency 15 in the Europarl Training set (5,080 English words and 6,878 Italian words), representing almost 3% of the whole text. Since WAGS is focused on OOV/rare words, manual alignments are provided for these words only, and not for the whole sentences. Two off-the-shelf word aligners have been evaluated on WAGS, and results have been compared to those obtained on an existing benchmark tailored to full text alignment. The results obtained confirm that WAGS is a valuable resource, which allows a statistically sound evaluation of WA systems’ performance on OOV and rare words, as well as extensive data analyses. WAGS is publicly released under a Creative Commons Attribution license. Keywords: Word Alignment, Gold Standard dataset, evaluation, out-of-vocabulary words, rare words 1. Introduction The task of Word Alignment (WA) consists of finding the correspondence between words that are translation of each other in a bilingual sentence pair (Brown et al., 1990). WA is a basic component of Statistical Machine Transla- tion (Och and Ney, 2004; Fraser and Marcu, 2007), but also other applications rely on WA, such as extraction of bilingual lexica (Smadja et al., 1996), word sense disam- biguation (Diab and Resnik, 2002), projection of linguistic information between languages (Yarowsky and Ngai, 2001; Kuhn, 2004; Bentivogli and Pianta, 2005). WA gold standards represent a crucial resource to evalu- ate and analyse WA systems’ performance, and nowadays various benchmarks for different language pairs are avail- able (Melamed, 1998; Och and Ney, 2000; Mihalcea and Pedersen, 2003; Lambert et al., 2005; Martin et al., 2005; Kruijff-Korbayov´a et al., 2006; Grac¸a et al., 2008; Macken, 2010; Holmqvist and Ahrenberg, 2011). Besides the lan- guages addressed, existing benchmarks differ in various respects – also depending on the final application to be evaluated – such as the parallel data used, the annotation scheme adopted (and related guidelines), the selection of words to be manually aligned. Regarding this latter is- sue, two main approaches were followed in previous works: full text alignment, where all words in the text are manu- ally aligned, and sample word alignment, where a set of test words are selected and only those words are manually aligned (V´eronis and Langlais, 2000; Merkel, 1999; Ahren- berg et al., 2002). One of the most challenging issues for current state-of- the-art word aligners is that they show poor generalization capability and are prone to errors when infrequent …

Content truncated to 500 words