Hypotheses Selection Criteria in a Reranking Framework for Spoken Language Understanding

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1104–1115, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Hypotheses Selection Criteria in a Reranking Framework for Spoken Language Understanding Marco Dinarelli LIMSI-CNRS B.P. 133, 91403 Orsay Cedex France marcod@limsi.fr Sophie Rosset LIMSI-CNRS B.P. 133, 91403 Orsay Cedex France rosset@limsi.fr Abstract Reranking models have been successfully ap- plied to many tasks of Natural Language Pro- cessing. However, there are two aspects of this approach that need a deeper investiga- tion: (i) Assessment of hypotheses generated for reranking at classification phase: baseline models generate a list of hypotheses and these are used for reranking without any assess- ment; (ii) Detection of cases where rerank- ing models provide a worst result: the best hypothesis provided by the reranking model is assumed to be always the best result. In some cases the reranking model provides an incorrect hypothesis while the baseline best hypothesis is correct, especially when base- line models are accurate. In this paper we propose solutions for these two aspects: (i) a semantic inconsistency metric to select pos- sibly more correct n-best hypotheses, from a large set generated by an SLU basiline model. The selected hypotheses are reranked apply- ing a state-of-the-art model based on Partial Tree Kernels, which encode SLU hypothe- ses in Support Vector Machines with com- plex structured features; (ii) finally, we apply a decision strategy, based on confidence val- ues, to select the final hypothesis between the first ranked hypothesis provided by the base- line SLU model and the first ranked hypothe- sis provided by the re-ranker. We show the ef- fectiveness of these solutions presenting com- parative results obtained reranking hypothe- ses generated by a very accurate Conditional Random Field model. We evaluate our ap- proach on the French MEDIA corpus. The re- sults show significant improvements with re- spect to current state-of-the-art and previous re-ranking models. 1 Introduction Discriminative reranking is a widely used approach for several Natural Language Processing (NLP) tasks: Syntactic Parsing (Collins, 2000), Named En- tity Recognition (Collins, 2000; Collins and Duffy, 2001), Semantic Role Labelling (Moschitti et al., 2008), Machine Translation (Shen et al., 2004), Question Answering (Moschitti et al., 2007). Re- cently reranking approaches have been successfully applied also to Spoken Language Understanding (SLU) (Dinarelli et al., 2009b). Discriminative Reranking combines two models: a first SLU model is used to generate a ranked list of n-best hypotheses; a reranking model sorts the list based on a different score and the final result is the new top ranked hypothesis. The advantage of reranking approaches is in the possibility to learn di- rectly complex dependencies in the output domain, as this is provided in the hypotheses generated by the baseline model. In previous approaches complex features are ex- tracted from the hypotheses for both training and classification phase, but there are very few stud- ies on approaches that can be applied to search in the hypotheses space generated by the baseline SLU model. Moreover, to keep …

Content truncated to 500 words