Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7755–7765, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7755 Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning Xiaoxiao Guo⇤ IBM Research xiaoxiao.guo@ibm.com Mo Yu⇤ IBM Research yum@us.ibm.com Yupeng Gao IBM Research yupeng.gao@ibm.com Chuang Gan MIT-IBM Watson AI Lab chuangg@ibm.com Murray Campbell IBM Research mcam@us.ibm.com Shiyu Chang MIT-IBM Watson AI Lab shiyu.chang@ibm.com Abstract Interactive Fiction (IF) games with real human- written natural language texts provide a new natural evaluation for language understanding techniques. In contrast to previous text games with mostly synthetic texts, IF games pose lan- guage understanding challenges on the human- written textual descriptions of diverse and so- phisticated game worlds and language genera- tion challenges on the action command gener- ation from less restricted combinatorial space. We take a novel perspective of IF game solv- ing and re-formulate it as Multi-Passage Read- ing Comprehension (MPRC) tasks. Our ap- proaches utilize the context-query attention mechanisms and the structured prediction in MPRC to efficiently generate and evaluate ac- tion outputs and apply an object-centric his- torical observation retrieval strategy to miti- gate the partial observability of the textual ob- servations. Extensive experiments on the re- cent IF benchmark (Jericho) demonstrate clear advantages of our approaches achieving high winning rates and low data requirements com- pared to all previous approaches.1 1 Introduction Interactive systems capable of understanding natu- ral language and responding in the form of natural language text have high potentials in various appli- cations. In pursuit of building and evaluating such systems, we study learning agents for Interactive Fiction (IF) games. IF games are world-simulating software in which players use text commands to control the protagonist and influence the world, as illustrated in Figure 1. IF gameplay agents need to simultaneously understand the game’s informa- tion from a text display (observation) and generate ⇤ Primary authors. 1Source code is available at: https://github.com/ XiaoxiaoGuo/rcdqn. Figure 1: Sample gameplay for the classic dungeon game Zork1. The objective is to solve various puzzles and collect the 19 treasures to install the trophy case. The player receives textual observations describing the current game state and ad- ditional reward scalars encoding the game designers’ objective of game progress. The player sends textual action commands to control the protagonist. natural language command (action) via a text in- put interface. Without providing an explicit game strategy, the agents need to identify behaviors that maximize objective-encoded cumulative rewards. IF games composed of human-written texts (dis- tinct from previous text games with synthetic texts) create superb new opportunities for studying and evaluating natural language understanding (NLU) techniques due to their unique characteristics. (1) Game designers elaborately craft on the literari- ness of the narrative texts to attract players when creating IF games. The resulted texts in IF games are more linguistically diverse and sophisticated than the template-generated ones in synthetic text games. (2) The language contexts of IF games 7756 … Object-based past observation retrieval Object extraction RC-model based observation-action value …

Content truncated to 500 words