Evaluating (and Improving) Sentence Alignment under Noisy Conditions

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 484–493, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Evaluating (and Improving) Sentence Alignment under Noisy Conditions Omar Zaidan Microsoft Research, USA ozaidan@cs.jhu.edu Vishal Chowdhary Microsoft Research, USA vishalc@microsoft.com Abstract Sentence alignment is an important step in the preparation of parallel data. Most aligners do not perform very well when the input is a noisy, rather than a highly- parallel, document pair. Evaluating align- ers under noisy conditions would seem to require creating an evaluation dataset by manually annotating a noisy document for gold-standard alignments. Such a costly process hinders our ability to evaluate an aligner under various types and lev- els of noise. In this paper, we propose a new evaluation framework for sentence aligners, which is particularly suitable for noisy-data evaluation. Our approach is unique as it requires no manual labeling, instead relying on small parallel datasets (already at the disposal of MT researchers) to generate many evaluation datasets that mimic a variety of noisy conditions. We use our framework to perform a compre- hensive comparison of three aligners un- der noisy conditions. Furthermore, our framework facilitates the fine-tuning of a state-of-the-art sentence aligner, allowing us to substantially increase its recall rates by anywhere from 5% to 14% (absolute) across several language pairs. 1 Introduction Virtually all training pipelines of statistical ma- chine translation systems expect training data to be in the form of a sequence of parallel sentence pairs. This means that a pair of parallel documents must first be segmented into a sequence of aligned sentence pairs, discarding or combining sentences when needed, and aligning sentences as appropri- ate. The performance and output of an SMT sys- tem is directly dependent on the amount and qual- ity of available training data. Therefore, it is crit- ical to perform this sentence alignment step prop- erly, ensuring both high recall (to have as much training data as possible) and high precision (to avoid noisy training data). While sentence aligners achieve excellent per- formance on highly-parallel, clean data, the task is much more difficult under noisy conditions. Some prior work has investigated evaluation under noisy conditions (see section 6), but the major focus of prior work has been the clean-data scenario, where accuracy rates exceed 98% (e.g. Simard et al. (1993), Moore (2002)). For one thing, this meant that the various sentence alignment algorithms dif- fer only slightly in absolute terms. Similarly, fine- tuning any one of those algorithms might not seem to have an impact on performance. More impor- tantly, this also meant that we do not have a clear understanding of how well these algorithms would perform under noisy conditions. Arguably, there was little need to examine sen- tence alignment of noisy datasets in early MT re- search, since almost all training data came from high-quality, highly-parallel sources, such as UN documents or parliamentary proceedings.1 How- ever, recent efforts have attempted to utilize web resources and non-perfectly-parallel texts, such as Wikipedia articles and news stories …

Content truncated to 500 words