Category-Based Strategy-Driven Question Generator for Visual Dialogue

Category-Based Strategy-Driven Question Generator for Visual Dialogue Yanan Shi1, Yanxin Tan1, Fangxiang Feng1, Chunping Zheng1 and Xiaojie Wang1� 1Beijing University of Posts and Telecommunications, Beijing, China apilis@bupt.edu.cn tobytyx@gmail.com {fxfeng, zhengchunping, xjwang}@bupt.edu.cn Abstract GuessWhat?! is a task-oriented visual dialogue task which has two players, a guesser and an oracle. Guesser aims to locate the object supposed by oracle by asking several Yes/No questions which are answered by oracle. How to ask proper questions is crucial to achieve the final goal of the whole task. Previous methods generally use an word-level generator, which is hard to grasp the dialogue-level questioning strategy. They often generate repeated or useless questions. This paper proposes a sentence-level category-based strategy-driven question generator(CSQG) to explicitly provide a category based questioning strategy for the generator. First we encode the image and the dialogue history to decide the category of the next question to be generated. Then the question is generated with the helps of category-based dialogue strategy as well as encoding of both the image and dialogue history. The evaluation on large-scale visual dialogue dataset GuessWhat?! shows that our method can help guesser achieve 51.71% success rate which is the state-of-the-art on the supervised training methods. 1 Introduction Goal-oriented Visual Dialogue is one of the multi-modal task which has gained increasing attentions. It can usually be explained as a task which using a couple of visual related turns of communication to reach a specific goal, such as GuessWhich (Chattopadhyay et al., 2017), VisDial (Das et al., 2017), Multimodal dialogs (MMD) (Saha et al., 2018), GuessWhat?! (de Vries et al., 2017) and so-on. GuessWhat?! (de Vries et al., 2017) is a two-player game which can be named guesser and oracle. Given an image, the oracle choose an object in the image without telling the guesser. Guesser reads the image and asks several Yes/No questions, at the meanwhile Oracle uses ’Yes’, ’No’ and ’N/A’ to answer them. After a fixed turns of questioning, Guesser should guess the chosen object in image and one game finishes. As is mentioned above, there are two sub tasks: one is ask question(QGen), and the other is Guess. Figure 1 gives a sample in GuessWhat?! dataset. To reach the goal of the entire game, the agent for task QGen should be able to make use of informa- tion from both the dialogue history and the image. de Vries et al. (de Vries et al., 2017) use an encoder- decoder model to generate questions, which is the first QGen model. After that, some researchers manage to improve the generation quality by attention mechanism. Zhuang et al. (Zhuang et al., 2018) proposed a parallel attention (PLAN) network. Deng et al. (Deng et al., 2018) proposed accumulated attention mechanism (A-ATT). Xu et al. (Xu et al., 2020) focused on boosting the agent to generate highly effi- cient questions and obtains reliable visual attentions and proposed Answer-Driven Visual State Estimator (ADVSE). On the other hand, Strub et al. (Strub et al., 2017) introduced reinforcement learning (RL). Pang et al. (Pang …

Content truncated to 500 words