Learning to translate with products of novices: a suite of open-ended challenge problems for teaching MT

Transactions of the Association for Computational Linguistics, 1 (2013) 165–178. Action Editor: David Chiang. Submitted 11/2012; Revised 3/2013; Published 5/2013. c⃝2013 Association for Computational Linguistics. Learning to translate with products of novices: a suite of open-ended challenge problems for teaching MT Adam Lopez1, Matt Post1, Chris Callison-Burch1,2, Jonathan Weese, Juri Ganitkevitch, Narges Ahmidi, Olivia Buzek, Leah Hanson, Beenish Jamil, Matthias Lee, Ya-Ting Lin, Henry Pao, Fatima Rivera, Leili Shahriyari, Debu Sinha, Adam Teichert, Stephen Wampler, Michael Weinberger, Daguang Xu, Lin Yang, and Shang Zhao∗ Department of Computer Science, Johns Hopkins University 1Human Language Technology Center of Excellence, Johns Hopkins University 2Computer and Information Science Department, University of Pennsylvania Abstract Machine translation (MT) draws from several different disciplines, making it a complex sub- ject to teach. There are excellent pedagogical texts, but problems in MT and current algo- rithms for solving them are best learned by doing. As a centerpiece of our MT course, we devised a series of open-ended challenges for students in which the goal was to im- prove performance on carefully constrained instances of four key MT tasks: alignment, decoding, evaluation, and reranking. Students brought a diverse set of techniques to the prob- lems, including some novel solutions which performed remarkably well. A surprising and exciting outcome was that student solutions or their combinations fared competitively on some tasks, demonstrating that even newcom- ers to the field can help improve the state-of- the-art on hard NLP problems while simulta- neously learning a great deal. The problems, baseline code, and results are freely available. 1 Introduction A decade ago, students interested in natural lan- guage processing arrived at universities having been exposed to the idea of machine translation (MT) primarily through science fiction. Today, incoming students have been exposed to services like Google Translate since they were in secondary school or ear- lier. For them, MT is science fact. So it makes sense to teach statistical MT, either on its own or as a unit ∗ The first five authors were instructors and the remaining au- thors were students in the worked described here. This research was conducted while Chris Callison-Burch was at Johns Hop- kins University. in a class on natural language processing (NLP), ma- chine learning (ML), or artificial intelligence (AI). A course that promises to show students how Google Translate works and teach them how to build some- thing like it is especially appealing, and several uni- versities and summer schools now offer such classes. There are excellent introductory texts—depending on the level of detail required, instructors can choose from a comprehensive MT textbook (Koehn, 2010), a chapter of a popular NLP textbook (Jurafsky and Martin, 2009), a tutorial survey (Lopez, 2008), or an intuitive tutorial on the IBM Models (Knight, 1999b), among many others. But MT is not just an object of academic study. It’s a real application that isn’t fully perfected, and the best way to learn about it is to build an MT sys- tem. This can be done with open-source toolkits such …

Content truncated to 500 words