Understand before Answer: Improve Temporal Reading Comprehension via Precise Question Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 12 - 37 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning Natural Language Generation with Truncated Reinforcement Learning Alice Martin Samovar, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau Guillaume Quispe and Charles Ollion CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau Sylvain Le Corff Samovar, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau Florian Strub Deep Mind Olivier Pietquin Google Brain Abstract This paper introduces TRUncated ReinForcement Learning for Language (TrufLL), an original approach to train conditional language models without a supervised learning phase, by only us- ing reinforcement learning (RL). As RL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary space using a generic language model. TrufLL thus enables to train a language agent by solely interacting with its environment without any task-specific prior knowledge; it is only guided with a task-agnostic language model. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pretrained policy flaws such as language or exposure biases. We evaluate TrufLL on two visual question generation tasks, for which we report positive results over performance and language metrics, which we then corroborate with a human evaluation. To our knowledge, it is the first approach that successfully learns a language generation policy without pre-training, using only reinforcement learning. 1 1 Introduction Since the development of generic language models trained on massive unlabelled text corpora (Radford et al., 2019; Brown et al., 2020), state-of-the art language processing systems rely on sequential transfer learning (Ruder, 2019). The pretrained Language Model (LM) is fine-tuned on the down- stream task using a standard supervised learning (SL) 1Code is available at https://github.com/ AMDonati/RL-NLP Agent VQA model carboy tall the Truncation run Language Model What is the tall boy holding ? What is the ... Bat Bat r=1 Agent Truncation with LM Bat Figure 1: (left) In a conditional language generation task as VQG, TrufLL truncates the vocabulary space by using a language model. Here, ’run,’ and ’the’ are syntactically incorrect and thus truncated. Yet, ’car’ is not trimmed as the LM is not visually grounded. (right) In a VQG training loop, the agent generates a question given an image-answer pair, which is then fed to a VQA model predicting an expected answer. If both answers match, the agent is rewarded. objective (Wu et al., 2019; Peters et al., 2019). Yet, such an approach suffers from several issues (Chen et al., 2020): (i) catastrophic forgetting when a model forgets previously learned knowledge and overfits to target domains, (ii) computational inefficiency from fine-tuning billion-parameters networks, and (iii) the need of supervised datasets. Moreover, task-specific language models learned with SL suffer from well-studied text degeneration issues (Holtzman et al., 2019), such as the exposure bias (Bengio et al., 2015), language biases (Saleh et al., 2020; Jaques et al., 2020), or a lack of diversity (Li et al., 2015). On …

Content truncated to 500 words