An Evaluation of Statistical Post-Editing Systems Applied to RBMT and SMT Systems

Proceedings of COLING 2012: Technical Papers, pages 215–230, COLING 2012, Mumbai, December 2012. An Evaluation of Statistical Post-editing Systems Applied to RBMT and SMT Systems Hanna Béchara1,2 Raphaël Rubino1 Y i f an He3 Yanjun Ma4 Jose f van Genabith1,2 (1) NCLT, School of Computing, DCU, Dublin, Ireland (2) CNGL, School of Computing, DCU, Dublin, Ireland (3) CIMS, NYU, New York, USA (4) Baidu, Beijing, China hbechara@computing.dcu.ie, raphael.rubino@computing.dcu.ie, yhe@cs.nyu.edu, yma@baidu.com, josef@computing.dcu.ie ABSTRACT Statistical post-editing (SPE) of the output produced by rule-based MT (RBMT) systems has been reported to produce extraordinary BLEU (and other automatic evaluation) score improvements. SPE has also been applied to the output of statistical MT (SMT) systems, albeit with more mixed results. We present a statistical post-editing pipeline and evaluate the outputs using automatic and human evaluation techniques, comparing the two SPE pipeline systems (RBMT + SPE and SMT + SPE) with the pure RBMT and SMT system, in an SPE scenario that uses independently existing bitext data, rather than manually corrected first stage MT output, as its training data. Our results show that although automatic evaluation metrics favour the pure SMT system, human evaluators prefer the output provided by the statistically post-edited RBMT system. KEYWORDS: Rule-Based Machine Translation, Statistical Machine Translation, Statistical Post- Editing. 215 1 Introduction Human evaluation is a core component of shared tasks such as WMT, and is often considered the gold standard in evaluation of translation systems. Automatic evaluation metrics, however, are much less costly, much more time efficient and enable automatic tuning of SMT system parameters (e.g. using MERT), which may require a number of iterations to converge. Statistical post-editing of the output of RBMT and SMT systems is an active field of research and RBMT + SPE pipelines are by now a commercial reality. 1 Automatic post-editing of rule-based machine translation systems (Simard et al., 2007; Terumasa, 2007; Kuhn et al., 2010) has shown (in some cases) spectacular improvements in translation quality measured in terms of automatic evaluation scores. Furthermore, SPE has also been applied to the output of statistical MT (SMT) systems (Oflazer and El-Khalout, 2007; Potet et al., 2011; Béchara et al., 2011; Rubino et al., 2012), albeit with more mixed results. However, to date, despite considerable interest in the area, the comparison between SPE pipelines and pure SMT and RBMT systems is not fully researched. Previous research can be categorised into roughly two classes: in one approach (Simard et al., 2007; Dugast et al., 2007; Potet et al., 2011), manually corrected (i.e. post-edited) MT output is used as the target side for training the SPE system (i.e. a "mono-lingual" SMT system trained on the output of the first stage MT system as source and the manually corrected first stage MT output as target, and then applied to the output of the first stage MT system on unseen source side input data), while the other approach (Oflazer and El-Khalout, 2007; Béchara et al., 2011; Rubino et al., 2012) simply uses available bi-text training data (such as translation memories …

Content truncated to 500 words