A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4111–4124 August 1–6, 2021. ©2021 Association for Computational Linguistics 4111 A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering Zhihong Shao1, Lifeng Shang2, Qun Liu2, Minlie Huang1∗ 1The CoAI group, DCST, Tsinghua University, Institute for Artificial Intelligence; 1State Key Lab of Intelligent Technology and Systems; 1Beijing National Research Center for Information Science and Technology; 1Tsinghua University, Beijing 100084, China 2Huawei Noah’s Ark Lab szh19@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn {shang.lifeng, qun.liu}@huawei.com Abstract Weakly supervised question answering usually has only the final answers as supervision sig- nals while the correct solutions to derive the answers are not provided. This setting gives rise to the spurious solution problem: there may exist many spurious solutions that coinci- dentally derive the correct answer, but training on such solutions can hurt model performance (e.g., producing wrong solutions or answers). For example, for discrete reasoning tasks as on DROP, there may exist many equations to de- rive a numeric answer, and typically only one of them is correct. Previous learning meth- ods mostly filter out spurious solutions with heuristics or using model confidence, but do not explicitly exploit the semantic correlations between a question and its solution. In this pa- per, to alleviate the spurious solution problem, we propose to explicitly exploit such seman- tic correlations by maximizing the mutual in- formation between question-answer pairs and predicted solutions. Extensive experiments on four question answering datasets show that our method significantly outperforms previous learning methods in terms of task performance and is more effective in training models to pro- duce correct solutions. 1 Introduction Weakly supervised question answering is a com- mon setting of question answering (QA) where only final answers are provided as supervision sig- nals while the correct solutions to derive them are not. This setting simplifies data collection, but ex- poses model learning to the spurious solution prob- lem: there may exist many spurious ways to derive the correct answer, and training a model with spu- rious solutions can hurt model performance (e.g., misleading the model to produce unreasonable so- lutions or wrong answers). As shown in Fig 1, ∗*Corresponding author: Minlie Huang. Multi-mention Reading Comprehension Question: In the television series ‘Thunderbirds’, what is Lady Penelope’s surname? Answer: Creighton Ward Document(s): Born on 24 December 2039, Lady Penelope is the 26-year old daughter of aristocrat Lord Hugh Creighton Ward and his wife, Amelia. The early years of her life were spent at Creighton Ward Mansion. … Lady Penelope Creighton Ward is a fictional character introduced in the British mid-1960s Supermarionation television series Thunderbirds, … Perce is the gardener for the 2000 acre Creighton Ward estate and a friend of Parker. … Possible Solution(s): ``Creighton Ward’’ across the document(s), only the third one is correct Discrete Reasoning over Paragraphs Question: How many years after the Battle of Powder River did Powerville Montana become the first establishment in the county? Answer: …

Content truncated to 500 words