Reasoning Over Semantic-Level Graph for Fact Checking
37422Authors
nan
Abstract
AbstractFact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.
Keywords
Full Content
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 583–592 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 583 End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2 Donghoon Ham1∗, Jeong-Gwan Lee1∗, Youngsoo Jang1, Kee-Eung Kim1,2 1School of Computing, KAIST, Daejeon, Republic of Korea 2Graduate School of AI, KAIST, Daejeon, Republic of Korea {dhham, jglee, ysjang}@ai.kaist.ac.kr, kekim@kaist.ac.kr Abstract The goal-oriented dialogue system needs to be optimized for tracking the dialogue flow and carrying out an effective conversation un- der various situations to meet the user goal. The traditional approach to building such a dialogue system is to take a pipelined mod- ular architecture, where its modules are op- timized individually. However, such an op- timization scheme does not necessarily yield an overall performance improvement of the whole system. On the other hand, end-to-end dialogue systems with monolithic neural archi- tecture are often trained only with input-output utterances, without taking into account the en- tire annotations available in the corpus. This scheme makes it difficult for goal-oriented di- alogues where the system needs to be inte- grated with external systems or to provide in- terpretable information about why the system generated a particular response. In this paper, we present an end-to-end neural architecture for dialogue systems that addresses both chal- lenges above. Our dialogue system achieved the success rate of 68.32%, the language un- derstanding score of 4.149, and the response appropriateness score of 4.287 in human eval- uations, which ranked the system at the top po- sition in the end-to-end multi-domain dialogue system task in the 8th dialogue systems tech- nology challenge (DSTC8). 1 Introduction The goal-oriented dialogue system helps users achieve their goals such as requesting information or executing commands via natural language con- versations. It is thus crucial for the dialogue system to keep track of the dialogue flow and carry out an effective conversation, even when the user goal is complicated or the dialogue flow is suddenly changed. ∗ : Equal contribution The traditional approach to building a goal- oriented dialogue system mostly adopts a pipelined modular architecture, with the natural language un- derstanding (NLU) module (Kim et al., 2017; Lee et al., 2019b) that first recognizes and comprehends user’s intent and extracts values for slots, then the dialogue state tracking (DST) module (Williams et al., 2013) that tracks the values of slots, then the dialogue policy (POL) module that decides the system action, and then finally the natural lan- guage generation (NLG) module (Wen et al., 2015) that generates the utterance that corresponds to the system action. In some cases, multiple mod- ules are combined together, e.g. the Word-level DST (Ramadan et al., 2018; Wu et al., 2019; Lee et al., 2019a) which maps the dialogue history to the dialogue state (the composite function of NLU and DST), and the Word-level POL (Budzianowski et al., 2018; Pei et al., 2019; Chen et al., 2019; Mehri et al., 2019; Zhao et al., 2019) which maps the previous utterance and dialogue state …
Content truncated to 500 words
Labeled Text
AbstractFact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.
Publication Details
- Paper ID:
- 37422
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan