TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems
7659Authors
nan
Abstract
AbstractWe present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pre-trained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to “make sense” 86.5% of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model’s API call predictions were rated to be correct 93.9% of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. To facilitate future work on transaction-based dialog systems, we are publicly releasing the TicketTalk dataset at https://git.io/JL8an.
Keywords
Full Content
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 59–68 August 1–6, 2021. ©2021 Association for Computational Linguistics 59 Unified Dual-view Cognitive Model for Interpretable Claim Verification Lianwei Wu, Yuan Rao, Yuqian Lan, Ling Sun, Zhaoyin Qi Xi’an Key Lab. of Social Intelligence and Complexity Data Processing, School of Software Engineering, Xi’an Jiaotong University, China Shaanxi Joint Key Laboratory for Artifact Intelligence(Sub-Lab of Xi’an Jiaotong University), China Research Institute of Xi’an Jiaotong University, Shenzhen, China {stayhungry,yuqian lan xjtu,sunling,daqige123}@stu.xjtu.edu.cn raoyuan@mail.xjtu.edu.cn Abstract Recent studies constructing direct interactions be- tween the claim and each single user response to capture evidence have shown remarkable suc- cess in interpretable claim verification. Owing to different single responses convey different cog- nition of individual users, the captured evidence belongs to the perspective of individual cognition. However, individuals’ cognition of social things is not always able to truly reflect the objective. There may be one-sided or biased semantics in their opinions on a claim. The captured evidence correspondingly contains some unobjective and biased information. In this paper, we propose a Dual-view model based on the views of Collective and Individual Cognition (CICD) for interpretable claim verification. For collective cognition, we not only capture the word-level semantics based on individual users, but also focus on sentence- level semantics (i.e., the overall responses) among all users to generate global evidence. For indi- vidual cognition, we select the top-k articles with high degree of difference and interact with the claim to explore the local key evidence fragments. To weaken the bias of individual cognition-view evidence, we devise an inconsistent loss to sup- press the divergence between global and local ev- idence for strengthening the consistent shared ev- idence between the both. Experiments on three benchmark datasets confirm the effectiveness of CICD. 1 Introduction The problem of claim credibility has seriously affected the media ecosystem. Research (Allen et al., 2020) illus- trates that the prevalence of ‘fake news’ has decreased trust in public institutions, and undermined democracy. Meanwhile, ‘massive infodemic’ during COVID-19 has taken a great toll on health-care systems and lives (Fleming, 2020). Therefore, how to verify the claims spread in networks has become a crucial issue. Current approaches on claim verification could be divided into two categories: 1) The first category re- Claim: In such a hot and rainy season, we should pay attention to the prevention of dengue fever. Not only mosquitoes can transmit it, but it is said that the disease could also be transmitted through the air. R1: This year is really hard, Let's get over this summer soon! R2: I think it's true. My husband had dengue fever before, and I also got infected soon. Maybe he infected me. R3: No, not all types of mosquitoes transmit dengue fever. R4: False, please don't continue to spread. It has been refuted that dengue fever will spread in the air. such a hot and rainy season, we itoes can transmit it, but it is …
Content truncated to 500 words
Labeled Text
AbstractWe present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pre-trained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to “make sense” 86.5% of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model’s API call predictions were rated to be correct 93.9% of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. To facilitate future work on transaction-based dialog systems, we are publicly releasing the TicketTalk dataset at
Publication Details
- Paper ID:
- 7659
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan