Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3268 - 3283 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning Vivek Gupta1∗, Shuo Zhang2, Alakananda Vempala2, Yujie He2, Temma Choji2, Vivek Srikumar1 1University of Utah, 2Bloomberg, {vgupta, svivek}@cs.utah.edu, {szhang611, avempala, yhe247, tchoji}@bloomberg.net Abstract When pre-trained contextualized embedding- based models developed for unstructured data are adapted for structured tabular data, they perform admirably. However, recent probing studies show that these models use spurious correlations, and often predict inference labels by focusing on false evidence or ignoring it al- together. To study this issue, we introduce the task of Trustworthy Tabular Reasoning, where a model needs to extract evidence to be used for reasoning, in addition to predicting the label. As a case study, we propose a two- stage sequential prediction approach, which includes an evidence extraction and an infer- ence stage. First, we crowdsource evidence row labels and develop several unsupervised and supervised evidence extraction strategies for INFOTABS, a tabular NLI benchmark. Our evidence extraction strategy outperforms ear- lier baselines. On the downstream tabular in- ference task, using only the automatically ex- tracted evidence as the premise, our approach outperforms prior benchmarks. 1 Introduction Reasoning on tabular or semi-structured knowledge is a fundamental challenge for today’s natural lan- guage processing (NLP) systems. Two recently created tabular Natural language Inference (NLI) datasets, TabFact (Chen et al., 2020b) on Wikipedia relational tables and INFOTABS (Gupta et al., 2020) on Wikipedia Infoboxes help study the question of inferential reasoning over semi-structured tables. Today’s state-of-the-art for NLI over unstructured text uses contextualized embeddings (e.g., Devlin et al., 2019; Liu et al., 2019b). When adapted for tabular NLI by flattening tables into synthetic sentences using heuristics, these models achieve remarkable performance on the datasets. However, a recent study (Gupta et al., 2021) demonstrates that these models fail to reason prop- ∗Work done during an internship at Bloomberg Breakfast in America Relevant Released4 29 March 19794 H3 Recorded3,4 May-December 19783,4 H2, H3 Studio The Village Recorder in Los Angeles3 Genre Pop, Art Rock, Soft Rock Length2 46:062 H1 Label A&M Producer1 Peter Henderson, Super- tramp1 H1 H1: Supertramp produced1 an album that was less than an hour long2. H2: Most of Breakfast in America was recorded3 in the last month of 19783. H3: Breakfast in America was released4 the same month recording ended 4. Figure 1: A semi-structured premise (the table ‘Breakfast in America’) example from (Gupta et al., 2020). Hypotheses H1 are entailed by it, H2 is nei- ther entailed nor contradictory, and H3 is a contra- diction. The Relevant column shows the hypotheses that use the corresponding row. The colored text (and superscripts) in the table and hypothesis highlights relevance token level alignment. erly on the semi-structured inputs in many cases. For example, they can ignore relevant rows, and (a) focus on the irrelevant rows (Neeraja et al., 2021), (b) use only the hypothesis sentence (Poliak …

Content truncated to 500 words