Multi-Task Learning for Improved Discriminative Training in SMT

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 292–300, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Multi-Task Learning for Improved Discriminative Training in SMT Patrick Simianer and Stefan Riezler Department of Computational Linguistics Heidelberg University 69120 Heidelberg, Germany {simianer,riezler}@cl.uni-heidelberg.de Abstract Multi-task learning has been shown to be effective in various applications, including discriminative SMT. We present an exper- imental evaluation of the question whether multi-task learning depends on a “natu- ral” division of data into tasks that bal- ance shared and individual knowledge, or whether its inherent regularization makes multi-task learning a broadly applicable remedy against overfitting. To investi- gate this question, we compare “natural” tasks defined as sections of the Interna- tional Patent Classification versus “ran- dom” tasks defined as random shards in the context of patent SMT. We find that both versions of multi-task learning im- prove equally well over independent and pooled baselines, and gain nearly 2 BLEU points over standard MERT tuning. 1 Introduction Multi-task learning is motivated by situations where a number of statistical models need to be es- timated from data belonging to different tasks. It is assumed that the data are not completely indepen- dent of one another as they share some common- alities, yet they differ enough to counter a simple pooling of data. The goal of multi-task learning is to take advantage of commonalities among tasks by learning a shared model without neglecting in- dividual knowledge. For example, Obozinski et al. (2010) present an optical character recognition scenario where data consist of samples of hand- written characters from several writers. While the styles of different writers vary, it is expected that there are also commonalities on a pixel- or stroke- level that are shared across writers. Chapelle et al. (2011) present a scenario where data from search engine query logs are available for different coun- tries. While the rankings for some queries will have to be country-specific (they cite “football” as a query requiring different rankings in the US and the UK), a large fraction of queries will be country-insensitive. W¨aschle and Riezler (2012b) present multi-task learning for statistical machine translation (SMT) of patents from different classes (so-called sections) according to the International Patent Classification (IPC)1. While the vocabulary may differ between the different IPC sections, spe- cific legal jargon and a typical textual structure will be shared across IPC sections. As shown in the cited works, treating data from different writ- ers, countries, or IPC classes as data from differ- ent tasks, and applying generic multi-task learning to the specific scenario, improves learning results over learning independent or pooled models. The research question we ask in this paper is as follows: Is multi-task learning dependent on a “natural” task structure in the data, where shared and individual knowledge is properly balanced? Or can multi-task learning be seen as a general regularization technique that prevents overfitting irrespective of the task structure in the data? We investigate this research question on the ex- ample of discriminative training …

Content truncated to 500 words