Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 854–864, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation Michael Bloodgood Human Language Technology Center of Excellence Johns Hopkins University Baltimore, MD 21211 bloodgood@jhu.edu Chris Callison-Burch Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21211 ccb@cs.jhu.edu Abstract We explore how to improve machine trans- lation systems by adding more translation data in situations where we already have substantial resources. The main challenge is how to buck the trend of diminishing re- turns that is commonly encountered. We present an active learning-style data solic- itation algorithm to meet this challenge. We test it, gathering annotations via Ama- zon Mechanical Turk, and find that we get an order of magnitude increase in perfor- mance rates of improvement. 1 Introduction Figure 1 shows the learning curves for two state of the art statistical machine translation (SMT) sys- tems for Urdu-English translation. Observe how the learning curves rise rapidly at first but then a trend of diminishing returns occurs: put simply, the curves flatten. This paper investigates whether we can buck the trend of diminishing returns, and if so, how we can do it effectively. Active learning (AL) has been ap- plied to SMT recently (Haffari et al., 2009; Haffari and Sarkar, 2009) but they were interested in start- ing with a tiny seed set of data, and they stopped their investigations after only adding a relatively tiny amount of data as depicted in Figure 1. In contrast, we are interested in applying AL when a large amount of data already exists as is the case for many important lanuage pairs. We de- velop an AL algorithm that focuses on keeping an- notation costs (measured by time in seconds) low. It succeeds in doing this by only soliciting trans- lations for parts of sentences. We show that this gets a savings in human annotation time above and beyond what the reduction in # words annotated would have indicated by a factor of about three and speculate as to why. 0 2 4 6 8 x 1010 4 0 5 10 15 20 25 30 Number of Sentences in Training Data BLEU Score JSyntax and JHier Learning Curves on the LDC Urdu−English Language Pack (BLEU vs Sentences) jHier jSyntax as far as previous AL for SMT research studies were conducted where we begin our main investigations into bucking the trend of diminishing returns Figure 1: Syntax-based and Hierarchical Phrase- Based MT systems’ learning curves on the LDC Urdu-English language pack. The x-axis measures the number of sentence pairs in the training data. The y-axis measures BLEU score. Note the di- minishing returns as more data is added. Also note how relatively early on in the process pre- vious studies were terminated. In contrast, the focus of our main experiments doesn’t even be- gin until much higher performance has already been achieved with a period of diminishing …

Content truncated to 500 words