INFOTABS: Inference on Tables as Semi-structured Data

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2309 INFOTABS: Inference on Tables as Semi-structured Data Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, Vivek Srikumar School of Computing, University of Utah {vgupta,maitrey,pnokhiz,svivek}@cs.utah.edu Abstract In this paper, we observe that semi-structured tabulated text is ubiquitous; understanding them requires not only comprehending the meaning of text fragments, but also implicit re- lationships between them. We argue that such data can prove as a testing ground for under- standing how we reason about information. To study this, we introduce a new dataset called INFOTABS, comprising of human-written tex- tual hypotheses based on premises that are tables extracted from Wikipedia info-boxes. Our analysis shows that the semi-structured, multi-domain and heterogeneous nature of the premises admits complex, multi-faceted rea- soning. Experiments reveal that, while hu- man annotators agree on the relationships be- tween a table-hypothesis pair, several stan- dard modeling strategies are unsuccessful at the task, suggesting that reasoning about tables can pose a difficult modeling challenge. 1 Introduction Recent progress in text understanding has been driven by sophisticated neural networks based on contextual embeddings—e.g., BERT (Devlin et al., 2019), and its descendants—trained on massive datasets, such as SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and SQuAD (Ra- jpurkar et al., 2016). Several such models outper- form human baselines on these tasks on the bench- mark suites such as GLUE (Wang et al., 2019b). Reasoning about text requires a broad array of skills—making lexical inferences, interpreting the nuances of time and locations, and accounting for world knowledge and common sense. Have we achieved human-parity across such a diverse col- lection of reasoning skills? In this paper, we study this question by propos- ing an extension of the natural language inference (NLI) task (Dagan et al., 2005, and others). In Dressage Highest governing body International Federation for Equestrian Sports (FEI) Characteristics Contact No Team members Individual and team at inter- national levels Mixed gender Yes Equipment Horse, horse tack Venue Arena, indoor or outdoor Presence Country or region Worldwide Olympic 1912 Paralympic 1996 H1: Dressage was introduced in the Olympic games in 1912. H2: Both men and women compete in the equestrian sport of Dressage. H3: A dressage athlete can participate in both individual and team events. H4: FEI governs dressage only in the U.S. Figure 1: A semi-structured premise (the table). Two hypotheses (H1, H2) are entailed by it, H3 is neither entailed nor contradictory, and H4 is a contradiction. NLI, which asks whether a premise entails, contra- dicts or is unrelated to a hypothesis, the premise and the hypothesis are one or more sentences. Un- derstanding the premise requires understanding its linguistic structure and reasoning about it. We seek to separate these two components. Our work stems from the observation that we can make valid infer- ences about implicit information conveyed by the mere juxtaposition of snippets of text, as shown in the table describing Dressage …

Content truncated to 500 words