Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4182 - 4193 December 7-11, 2022 ©2022 Association for Computational Linguistics Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue Sunjae Yoon†, Eunseop Yoon†, Hee Suk Yoon†, Junyeong Kim‡, Chang D. Yoo†∗ †Korea Advanced Institute of Science and Technology (KAIST) ‡Chung-Ang University {sunjae.yoon,esyoon97,hskyoon,cd_yoo}@kaist.ac.kr; junyeongkim@cau.ac.kr Abstract Video-grounded Dialogue (VGD) aims to de- code an answer sentence to a question regard- ing a given video and dialogue context. De- spite the recent success of multi-modal reason- ing to generate answer sentences, existing di- alogue systems still suffer from a text halluci- nation problem, which denotes indiscriminate text-copying from input texts without an un- derstanding of the question. This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to over- lap with ground-truth texts. Hence, we de- sign Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucina- tion Regularization (THR) loss derived from the proposed information-theoretic text hal- lucination measurement approach. Applying THAM with current dialogue systems vali- dates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability. 1 Introduction Achieving a natural conversational agent that can do ‘look’ (i.e., understand what they are seeing) and ‘tell’ (i.e., converse what they are thinking) is desiderata in our vision-language community. By the broad application of conversational agent, it can potentially assist various subsections of our envi- ronment including education, entertainment, secu- rity, and visual or other impairments. For the natu- ral conversation between humans and computers, a video-grounded dialogue (VGD) task (Alamri et al., 2019; Hori et al., 2020) has been introduced to generate adequate conversational responses to the queries of humans, while following up on video and dialogue context, which gives more challenging than traditional image-grounded or text-grounded ∗Corresponding author dialogue tasks. To be specific, given video V , video caption C, dialogue history of past Q&A pairs: H = {(Q1, A1), ..., (Qr−1, Ar−1)}, and current r-th round question Qr, VGD system is expected to make free-form answer sentence Ar to given question. Despite recent advancements in multi- modal interactions including transformer (Vaswani et al., 2017), current VGD systems still suffer text hallucination problem, which denotes indiscrimi- nate text-copying from input texts (i.e., question, caption, and dialogue history) to decode answer to- kens, but the generated answer sentences are rather inadequate and not related to the question. This is because current VGD systems learn spurious correlations from the fact that many ground-truth answers in the dataset include partial input texts, thus they perform incorrect text-copy from input texts, namely text hallucination, even in answers where input texts are unnecessary. Figure 1 gives two indiscriminate text halluci- nating cases confounded by spurious correlations in VGD. As shown in Figure 1(a), for the given question ‘does he place the towel and clothes any- where?’, we human identify where the man …

Content truncated to 500 words