Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression

FeTaQA: Free-form Table Question Answering Linyong Nan1 Chiachun Hsieh3 Ziming Mao1 Xi Victoria Lin2∗ Neha Verma1 Rui Zhang4 Wojciech Kry´sci´nski2 Hailey Schoelkopf1 Riley Kong5 Xiangru Tang1 Mutethia Mutuma1 Ben Rosand1 Isabel Trindade1 Renusree Bandaru4 Jacob Cunningham4 Caiming Xiong2 Dragomir Radev1,2 1 Yale University, USA 2 Salesforce Research, USA 3 The University of Hong Kong, China 4 Penn State University, USA 5 Archbishop Mitty High School, USA {linyong.nan, ziming.mao}@yale.edu, hsiehcc@connect.hku.hk Abstract Existing table question answering datasets contain abundant factual questions that pri- marily evaluate a QA system’s comprehension of query and tabular data. However, restricted by their short-form answers, these datasets fail to include question–answer interactions that represent more advanced and naturally occurring information needs: questions that ask for reasoning and integration of infor- mation pieces retrieved from a structured knowledge source. To complement the exist- ing datasets and to reveal the challenging na- ture of the table-based question answering task, we introduce FeTaQA, a new dataset with 10K Wikipedia-based {table, question, free-form answer, supporting table cells} pairs. FeTaQA is collected from noteworthy de- scriptions of Wikipedia tables that contain information people tend to seek; generation of these descriptions requires advanced pro- cessing that humans perform on a daily basis: Understand the question and table, retrieve, integrate, infer, and conduct text planning and surface realization to generate an answer. We provide two benchmark methods for the proposed task: a pipeline method based on semantic parsing-based QA systems and an end-to-end method based on large pretrained text generation models, and show that FeTaQA poses a challenge for both methods. 1 Introduction Question Answering (QA) is the task of producing answers to natural language questions based on knowledge resources (Burke et al., 1997; Yao and Van Durme, 2014; Chen et al., 2017). One of the primary goals of QA is to allow users to directly and efficiently interact with large-scale and het- ∗Now at Facebook AI. erogeneous knowledge sources. In the real world, knowledge sources take a variety of forms, includ- ing unstructured texts (documents, passages, or conversations), structured knowledge bases, and semi-structured tables, each requiring dedicated modeling approaches. For QA over text, a sequence modeling ap- proach is usually adopted to encode the query and the context, and answers are either categori- cal (Lai et al., 2017), extractive (Rajpurkar et al., 2016; Yang et al., 2018), or abstractive/generative (Kocisk´y et al., 2017; Nguyen et al., 2016; Fan et al., 2019; Kwiatkowski et al., 2019). For QA over table, a common approach is to apply seman- tic parsing on the query and the table schema to generate a logical form (e.g., a SQL-like database query) that can be executed to retrieve the answer from the relevant portion of the table (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017; Yu et al., 2018). The answers are extracted facts/entities in the table, therefore usually in short-form. Though existing datasets have enabled sig- nificant progress for table QA, their limitations prevent them from reflecting the challenging na- ture of the task. The …

Content truncated to 500 words