Multi-Row, Multi-Span Distant Supervision For Table+Text Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 744–768 July 9-14, 2023 ©2023 Association for Computational Linguistics Multi-VALUE: A Framework for Cross-Dialectal English NLP Caleb Ziems William Held Jingfeng Yang Jwala Dhamala Rahul Gupta Diyi Yang Stanford University, Georgia Institute of Technology, Amazon {cziems, diyiy}@stanford.edu, {wheld3}@gatech.edu, {jddhamal, yjfllpyym, gupra}@amazon.com Abstract Dialect differences caused by regional, social, and economic factors cause performance dis- crepancies for many groups of language tech- nology users. Inclusive and equitable language technology must critically be dialect invari- ant, meaning that performance remains con- stant over dialectal shifts. Current systems of- ten fall short of this ideal since they are de- signed and tested on a single dialect: Standard American English (SAE). We introduce a suite of resources for evaluating and achieving En- glish dialect invariance. The resource is called Multi-VALUE, a controllable rule-based trans- lation system spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE maps SAE to synthetic forms of each dialect. First, we use this system to stress tests question answering, machine translation, and semantic parsing. Stress tests reveal significant perfor- mance disparities for leading models on non- standard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Fi- nally, we partner with native speakers of Chi- cano and Indian English to release new gold- standard variants of the popular CoQA task. To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see http://value-nlp.org/. 1 Introduction “[Often, speakers] will not be hampered by the lack of language technology in their local lan- guage, but by the lack of support for their variety of the contact language.” — Steven Bird (2022) Global contact languages like English will con- tinue to have an outsized impact on commerce, economics, wellbeing, and equity worldwide. En- glish, like any other language, is subject to variation Equal contribution. across time (Yang, 2000) and between speakers or speaker groups (Eckert, 2017; Holmes and Meyer- hoff, 2008). Rather than focusing on social status or political power (Stewart, 1968; Chambers and Trudgill, 1998), linguists define dialects as descrip- tive sets of correlated features common across a group of speakers (Nerbonne, 2009). Current pre- training paradigms employ content filters that can exclude text in English dialects other than Standard American and British (Gururangan et al., 2022), which leads to performance gaps for other varieties. These discrepancies in Natural Language Process- ing (NLP) cause allocational harms for dialectal speakers in downstream applications (Bender et al., 2021), making dialect robustness a critical need for fair and inclusive language technology. This disparity is clear in a growing body of em- pirical work on African American English (Ziems et al., 2022; Halevy et al., 2021; Blodgett et al., 2018; Jurgens et al., 2017; Kiritchenko and Mo- hammad, 2016). However, there does not yet ex- ist a systematic exploration of robustness across multiple Englishes, nor of models’ ability to trans- fer knowledge …

Content truncated to 500 words