Methods and Practical Issues in Evaluating Alignment Techniques

Methods and Practical Issues in Evaluating Alignment Techniques Philippe Langlais CTT/KTH SE-I0044 Stockholm CERI-LIA, AGROPARC BP 1228 F-84911 Avignon Cedex 9 Philippe.Langlais~speech.kth.se Michel Simard RALI-DIRO Univ. de Montrdal Qudbec, Canada H3C 3J7 shnardm~IRO.UMontreal.CA Jean Vdronis LPL, Univ. de Provence 29, Av. R. Schuman F-13621 Aix-en-Provence Cedex 1 veronis~univ-aix.fr Abstract This paper describes the work achieved in the first half of a 4-year cooperative research project (ARCADE), financed by AUPELF-UREF. The project is devoted to the evaluation of paral- lel text alignment techniques. In its first period ARCADE ran a competition between six sys- tems on a sentence-to-sentence alignment task which yielded two main types of results. First, a large reference bilingual corpus comprising of texts of different genres was created, each pre- senting various degrees of difficulty with respect to the alignment task. Second, significant methodological progress was made both on the evaluation protocols and metrics, and the algoritbm.q used by the dif- ferent systems. For the second phase, which is now underway, ARCADE has been opened to a larger number of teams who will tackle the problem of word-level alignment. 1 Introduction In the last few years, there has been a growing interest in parallel text alignment techniques. These techniques attempt to map various tex- tual units to their translation and have proven useful for a wide range of applicatious and tools. A simple example of such a tool is probably the TransSearch bilingual concordancing system (Isabelle et al., 1993), which allows a user to query a large archive of existing translations in order to find ready-made solutions to specific translation problems. Such a tool has proved ex- tremely useful not only for translators, but also for bilingual lexicographers (Langlois, 1996) and terminologists (Dagan and Church, 1994). More sophisticated applications based on alignment technology have also been the object of recent work, such as the automatic building of bilin- gual lexical resources (Melamed, 1996; Klavans and Tzoukermann, 1995), the automatic verifi- cation of translations (Macklovitch, 1995), the automatic dictation of translations (Brousseau et al., 1995) and even interactive machine trans- lation (Foster et al., 1997). Enthusiasm for this relatively new field was sparked early on by the apparent demonstra- tion that very simple techniques could yield al- most perfect results. For instance, to produce sentence alignments, Brown et al. (1991) and Gale and Church (1991) both proposed meth- ods that completely ignored the lexical content of the texts and both reported accuracy lev- els exceeding 98%. Unfortunately performance tends to deteriorate significantly when aligners are applied to corpora which are widely differ- ent from the training corpus, and/or where the alignments are not straightforward. For instance graphics, tables, "floating" notes and missing segments, which are very common in real texts, all result in a dramatic loss of efficiency. The truth is that, while text alignment is mostly an easy problem, especially when consid- ered at the sentence level, there are situations where even humans have a hard time making the right decision. In fact, it could be argued that, …

Content truncated to 500 words