Efficiency-based evaluation of aligners for industrial applications

Efficiency-based evaluation of aligners for industrial applications∗ Antonio Toral School of Computing Dublin City University Dublin, Ireland atoral@computing.dcu.ie Marc Poch IULA, Universitat Pompeu Fabra Barcelona, Spain marc.pochriera@upf.edu Pavel Pecina Faculty of Mathematics and Physics, Charles University Prague, Czech Republic pecina@ufal.mff.cuni.cz Gregor Thurmair Linguatec GmbH Munich, Germany g.thurmair@linguatec.de Abstract This paper presents a novel efficiency- based evaluation of sentence and word aligners. This assessment is critical in or- der to make a reliable use in industrial sce- narios. The evaluation shows that the re- sources required by aligners differ rather broadly. Subsequently, we establish lim- itation mechanisms on a set of aligners deployed as web services. These results, paired with the quality expected from the aligners, allow providers to choose the most appropriate aligner according to the task at hand. 1 Introduction Aligners refer in this paper to tools that, given a bilingual corpus, identify corresponding pairs of linguistic items, be they sentences (sentence align- ers) or words (word aligners). Alignment is a key component in corpus-based multilingual applica- tions. First, alignment is one of the most time- consuming tasks in building Machine Translation (MT) systems. In terms of quality, good align- ment is decisive for the final quality of the MT system; bad alignment decreases MT quality and inflates the phrase table with spurious translations with very low probabilities, which reduces system performance. Finally, for terminology acquisition, the choice of a good aligner determines whether the results of a term extraction tool are usable or not; alignment quality on phrase level differs from ∗We would like to thank Daniel Varga and Adrien Lardilleux for their feedback on Hunalign and Anymalign, respectively. We would like to thank Joachim Wagner for his help on using the cluster. This research has been partially funded by the EU project PANACEA (7FP-ITC-248064). ∗c⃝ 2012 European Association for Machine Translation. less than 5% (usable) to more than 40% (unusable) error rate (Aleksic and Thurmair, 2012). The performance of aligners is commonly eval- uated extrinsically, i.e. by measuring their im- pact in the result obtained by a MT system that uses the aligned corpus (Abdul-Rauf et al., 2010; Lardilleux and Lepage, 2009; Haghighi et al., 2009). Intrinsic evaluations have also been car- ried out, mainly by measuring the Alignment Error Rate (AER), precision and recall (von Waldenfels, 2006; Varga et al., 2005; Moore, 2002; Haghighi et al., 2009). Intrinsic evaluation is less popular due to two reasons (Fraser and Marcu, 2007): (i) it requires a gold standard and (ii) the correlation between AER and MT quality is very low. Both types of evaluation have, however, a common as- pect; they focus on measuring the quality of the output produced by aligners. Conversely, seldom if at all has it been considered to assess the ef- ficiency of aligners, i.e. to measure the compu- tational resources consumed (e.g. execution time, use of memory). However, this assessment is crit- ical if the aligners are to be exploited in an indus- trial scenario. This work is part of a wider project, …

Content truncated to 500 words