Speeding up Reinforcement Learning-based Information Extraction Training using Asynchronous Methods

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2658–2663 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Speeding up Reinforcement Learning-based Information Extraction Training using Asynchronous Methods Aditya Sharma Indian Institute of Science Bangalore, India adisharma075@gmail.com Zarana Parekh∗ DA-IICT Gandhinagar, India zaranaparekh17@gmail.com Partha Talukdar Indian Institute of Science Bangalore, India ppt@iisc.ac.in Abstract RLIE-DQN is a recently proposed Re- inforcement Learning-based Information Extraction (IE) technique which is able to incorporate external evidence during the extraction process. RLIE-DQN trains a single agent sequentially, training on one instance at a time. This results in sig- nificant training slowdown which is unde- sirable. We leverage recent advances in parallel RL training using asynchronous methods and propose RLIE-A3C. RLIE- A3C trains multiple agents in parallel and is able to achieve upto 6x training speedup over RLIE-DQN, while suffering no loss in average accuracy. 1 Introduction Extracting information about an event (or entity) involves multiple decisions, as one first needs to identify documents relevant to the event, extract relevant information from those documents, and finally reconcile various values obtained for the same relation of the event from different sources (Ahn, 2006). Search based methods for Infor- mation Extraction have been increasingly investi- gated (West et al., 2014); (Hegde and Talukdar, 2015); (Zhang et al., 2016); (Bing et al., 2017). (Kanani and McCallum, 2012) combine search and Information Extraction (IE) using Reinforce- ment Learning (RL), with the goal of selecting good actions while staying within resource con- straints, but don’t optimze for extraction accuracy. More recently, (Narasimhan et al., 2016) proposed a RL-based approach to model the IE process out- lined above. We shall refer to this approach as RLIE-DQN in this paper. RLIE-DQN trains an RL ∗Research carried out during an internship at the Indian Institute of Science, Bangalore. agent using Deep Q-Network (DQN) (Mnih et al., 2015) to select optimal actions to query for docu- ments and also reconcile extracted values. DQN trains a single agent sequentially, up- dating parameters based on one instance at a time. Each such instance is sampled from the en- tire training data, also called the experience re- play. Such sequential experience replay-based training results in slow learning, while requiring high memory and computation resources. In order to overcome this challenge, A3C (Asynchronous Advantage Actor-Critic), an asynchronous deep RL training algorithm, has been proposed recently (Mnih et al., 2016). A3C trains multiple RL agents in parallel, each of which estimates gradients lo- cally, and asynchronously updates globally shared parameters. Recent work has explored applica- tions for A3C in varied domains (Fernando et al., 2017), (Mirowski et al., 2016). In this paper, we propose RLIE-A3C which uses A3C-based parallel asynchronous agents for train- ing. This is in contrast to the sequential DQN training in RLIE-DQN. Differences between the training regimes of the two methods are shown in Figure 1. Through experiments on multiple real- world datasets, we find that RLIE-A3C achieves upto 6x training speedup compared to RLIE-DQN, while suffering no loss in accuracy. …

Content truncated to 500 words