Enhancing Tabular Reasoning with Pattern Exploiting Training

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 706–726 November 20–23, 2022. ©2022 Association for Computational Linguistics 706 Enhancing Tabular Reasoning with Pattern Exploiting Training Abhilash Reddy Shankarampeta1∗, Vivek Gupta2*†, Shuo Zhang3 1IIT Guwahati; 2University of Utah; 3Bloomberg sareddy53@gmail.com; vgupta@cs.utah.edu; szhang611@bloomberg.net Abstract Recent methods based on pre-trained language models have exhibited superior performance over tabular tasks (e.g., tabular NLI), despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs while reasoning over the tabu- lar data (Gupta et al., 2021). In this work, we utilize Pattern-Exploiting Training (PET) (i.e., strategic MLM) on pre-trained language models to strengthen these tabular reasoning models’ pre-existing knowledge and reasoning abilities. Our upgraded model exhibits a su- perior understanding of knowledge facts and tabular reasoning compared to current base- lines. Additionally, we demonstrate that such models are more effective for underlying down- stream tasks of tabular inference on INFOTABS. Furthermore, we show our model’s robustness against adversarial sets generated through vari- ous character and word level perturbations. 1 Introduction Natural Language Inference (NLI) is the problem of categorizing a hypothesis into entailment, con- tradiction, or neutral based on the given premise (Dagan et al., 2013). Large language models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019c) have been applied to large datasets like SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), where they have shown performance comparable to that of humans. However, the existing methods based on lan- guage models are ineffective for reasoning over semi-structured data (Gupta et al., 2021). These models often ignore relevant rows and use spurious correlations in hypothesis or pre-training informa- tion for making inferences (Neeraja et al., 2021; Poliak et al., 2018; Gururangan et al., 2018; Jain et al., 2021; Gupta et al., 2021). Due to existing biases in human curated datasets (Rajpurkar et al., ∗Equal Contribution †Corresponding Author Breakfast in America Released 29 March 1979 Recorded May–December 1978 Studio The Village Recorder in LA Genre Pop, art rock, soft rock Length 46:06 Label A&M Producer Peter Henderson, Supertramp H1: Breakfast in America is a pop album with a duration less than 50 minutes. H2: Peter Henderson produces only rock albums. H3: Breakfast in America was released towards the end of 1979. H4: Breakfast in America is recorded in California. H5: Supertramp is an English band. H6: The album was released on 29 March 1978. Table 1: An example of tabular premise from IN- FOTABS (Gupta et al., 2020). The hypotheses H1, H4 is entailed, H2, H5 is a neutral and H3, H6 is a con- tradiction. Here, the bold entries, which correspond to the first column, are the keys, while the corresponding entries in the second column of the same row are their respective values. 2018; Zhou and Bansal, 2020) with hypothesis hav- ing annotation artifacts (Gururangan et al., 2018), often models trained on …

Content truncated to 500 words