Deep Reinforcement Learning for Mention-Ranking Coreference Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Deep Reinforcement Learning for Mention-Ranking Coreference Models Kevin Clark Computer Science Department Stanford University kevclark@cs.stanford.edu Christopher D. Manning Computer Science Department Stanford University manning@cs.stanford.edu Abstract Coreference resolution systems are typically trained with heuristic loss functions that re- quire careful tuning. In this paper we in- stead apply reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics. We experi- ment with two approaches: the REINFORCE policy gradient algorithm and a reward- rescaled max-margin objective. We find the latter to be more effective, resulting in a sig- nificant improvement over the current state- of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task. 1 Introduction Coreference resolution systems typically operate by making sequences of local decisions (e.g., adding a coreference link between two mentions). How- ever, most measures of coreference resolution per- formance do not decompose over local decisions, which means the utility of a particular decision is not known until all other decisions have been made. Due to this difficulty, coreference systems are usually trained with loss functions that heuristically define the goodness of a particular coreference deci- sion. These losses contain hyperparameters that are carefully selected to ensure the model performs well according to coreference evaluation metrics. This complicates training, especially across different lan- guages and datasets where systems may work best with different settings of the hyperparameters. To address this, we explore using two variants of reinforcement learning to directly optimize a coref- erence system for coreference evaluation metrics. In particular, we modify the max-margin coreference objective proposed by Wiseman et al. (2015) by in- corporating the reward associated with each coref- erence decision into the loss’s slack rescaling. We also test the REINFORCE policy gradient algorithm (Williams, 1992). Our model is a neural mention-ranking model. Mention-ranking models score pairs of mentions for their likelihood of coreference rather than compar- ing partial coreference clusters. Hence they operate in a simple setting where coreference decisions are made independently. Although they are less expres- sive than entity-centric approaches to coreference (e.g., Haghighi and Klein, 2010), mention-ranking models are fast, scalable, and simple to train, caus- ing them to be the dominant approach to coreference in recent years (Durrett and Klein, 2013; Wiseman et al., 2015). Having independent actions is partic- ularly useful when applying reinforcement learning because it means a particular action’s effect on the final reward can be computed efficiently. We evaluate the models on the English and Chi- nese portions of the CoNLL 2012 Shared Task. The REINFORCE algorithm is competitive with a heuris- tic loss function while the reward-rescaled objective significantly outperforms both1. We attribute this to reward rescaling being well suited for a ranking task due to its max-margin loss as well as benefiting from directly optimizing for coreference metrics. Error analysis shows that using the reward-rescaling loss results in a similar number of mistakes as …

Content truncated to 500 words