CLAR: A Cross-Lingual Argument Regularizer for Semantic Role Labeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281–296 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 281 Understanding tables with intermediate pre-training Julian Martin Eisenschlos, Syrine Krichene, Thomas Müller Google Research, Zürich {eisenjulian,syrinekrichene,thomasmueller}@google.com Abstract Table entailment, the binary classification task of finding if a sentence is supported or re- futed by the content of a table, requires pars- ing language and table structure as well as nu- merical and discrete reasoning. While there is extensive work on textual entailment, ta- ble entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Moti- vated by the benefits of data augmentation, we create a balanced dataset of millions of auto- matically created training examples which are learned in an intermediate step prior to fine- tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT mod- els, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a mod- erate drop in accuracy. The different meth- ods set the new state-of-the-art on the TAB- FACT (Chen et al., 2020) and SQA datasets. 1 Introduction Textual entailment (Dagan et al., 2005), also known as natural language inference (Bowman et al., 2015), is a core natural language processing (NLP) task. It can predict effectiveness of reading com- prehension (Dagan et al., 2010), which argues that it can form the foundation of many other NLP tasks, and is a useful neural pre-training task (Subrama- nian et al., 2018; Conneau et al., 2017). Textual entailment is well studied, but many rele- vant data sources are structured or semi-structured: health data both worldwide and personal, fitness trackers, stock markets, and sport statistics. While some information needs can be anticipated by hand- crafted templates, user queries are often surprising, and having models that can reason and parse that structure can have a great impact in real world ap- plications (Khashabi et al., 2016; Clark, 2019). A recent example is TABFACT (Chen et al., 2020), a dataset of statements that are either en- tailed or refuted by tables from Wikipedia (Fig- ure 1). Because solving these entailment problems requires sophisticated reasoning and higher-order operations like arg max, averaging, or comparing, human accuracy remains substantially (18 points) ahead of the best models (Zhong et al., 2020). The current models are dominated by semantic parsing approaches that attempt to create logical forms from weak supervision. We, on the other hand, follow Herzig et al. (2020) and Chen et al. (2020) and encode the tables with BERT-based models to directly predict the entailment decision. But while BERT models for text have been scru- tinized and optimized for how to best pre-train and represent textual data, the same attention has not been applied to tabular data, limiting the ef- fectiveness in this setting. This paper addresses these …

Content truncated to 500 words