N-Best Reranking by Multitask Learning

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 375–383, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics N-best Reranking by Multitask Learning Kevin Duh Katsuhito Sudoh Hajime Tsukada Hideki Isozaki Masaaki Nagata NTT Communication Science Laboratories 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237, Japan {kevinduh,sudoh,tsukada,isozaki}@cslab.kecl.ntt.co.jp nagata.masaaki@lab.ntt.co.jp Abstract We propose a new framework for N-best reranking on sparse feature sets. The idea is to reformulate the reranking problem as a Multitask Learning problem, where each N-best list corresponds to a distinct task. This is motivated by the observation that N-best lists often show significant differ- ences in feature distributions. Training a single reranker directly on this heteroge- nous data can be difficult. Our proposed meta-algorithm solves this challenge by using multitask learning (such as ℓ1/ℓ2 regularization) to discover common feature representations across N- best lists. This meta-algorithm is simple to implement, and its modular approach al- lows one to plug-in different learning algo- rithms from existing literature. As a proof of concept, we show statistically signifi- cant improvements on a machine transla- tion system involving millions of features. 1 Introduction Many natural language processing applications, such as machine translation (MT), parsing, and language modeling, benefit from the N-best reranking framework (Shen et al., 2004; Collins and Koo, 2005; Roark et al., 2007). The advan- tage of N-best reranking is that it abstracts away the complexities of first-pass decoding, allowing the researcher to try new features and learning al- gorithms with fast experimental turnover. In the N-best reranking scenario, the training data consists of sets of hypotheses (i.e. N-best lists) generated by a first-pass system, along with their labels. Given a new N-best list, the goal is to rerank it such that the best hypothesis appears near the top of the list. Existing research have fo- cused on training a single reranker directly on the entire data. This approach is reasonable if the data is homogenous, but it fails when features vary sig- nificantly across different N-best lists. In partic- ular, when one employs sparse feature sets, one seldom finds features that are simultaneously ac- tive on multiple N-best lists. In this case, we believe it is more advantageous to view the N-best reranking problem as a multi- task learning problem, where each N-best list cor- responds to a distinct task. Multitask learning, a subfield of machine learning, focuses on how to effectively train on a set of different but related datasets (tasks). Our heterogenous N-best list data fits nicely with this assumption. The contribution of this work is three-fold: 1. We introduce the idea of viewing N-best reranking as a multitask learning problem. This view is particularly apt to any general reranking problem with sparse feature sets. 2. We propose a simple meta-algorithm that first discovers common feature representa- tions across N-bests (via multitask learning) before training a conventional reranker. Thus it is easily applicable to existing systems. 3. We demonstrate that our proposed method outperforms the conventional reranking ap- proach on a English-Japanese biomedical …

Content truncated to 500 words