MAVEN: A Massive General Domain Event Detection Dataset

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 163–170, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 163 More Bang for Your Buck: Natural Perturbation for Robust Question Answering Daniel Khashabi and Tushar Khot and Ashish Sabharwal Allen Institute for AI, Seattle, WA, U.S.A. {danielk,tushark,ashishs}@allenai.org Abstract Deep learning models for linguistic tasks re- quire large training datasets, which are expen- sive to create. As an alternative to the tradi- tional approach of creating new instances by repeating the process of creating one instance, we propose doing so by first collecting a set of seed examples and then applying human- driven natural perturbations (as opposed to rule-based machine perturbations), which of- ten change the gold label as well. Such per- turbations have the advantage of being rela- tively easier (and hence cheaper) to create than writing out completely new examples. Further, they help address the issue that even models achieving human-level scores on NLP datasets are known to be considerably sensitive to small changes in input. To evaluate the idea, we consider a recent question-answering dataset (BOOLQ) and study our approach as a function of the perturbation cost ratio, the relative cost of perturbing an existing question vs. creating a new one from scratch. We find that when nat- ural perturbations are moderately cheaper to create (cost ratio under 60%), it is more effec- tive to use them for training BOOLQ models: such models exhibit 9% higher robustness and 4.5% stronger generalization, while retaining performance on the original BOOLQ dataset. 1 Introduction Creating large datasets to train NLP models has become increasingly expensive. While many datasets (Bowman et al., 2015; Rajpurkar et al., 2016) targeting different linguistic tasks have been proposed, nearly all are created by repeating a fixed process used for writing a single example. This approach results in many independent examples, each generated from scratch. We propose an alter- native, often substantially cheaper training set con- struction method where, after collecting a few seed examples, the set is expanded by applying human- authored minimal perturbations to the seeds. D “Seed” Dataset “Perturbed” Dataset ◰ ◔ � ▲ ▼ ◀ ▶ ◰ ◱ ⬓ ◳ ◲ ◨ Related Paragraph: (Yonge Street) Provincial downloading separated Yonge Street from Highway 11 during the 1990s. As a result, Highway 11 does not start until Crown Hill just outside Barrie, several kilometres north of where the name ``Yonge Street'' ends. The Guinness Book of World Records no longer lists Yonge Street as the longest street in the world and has not chosen a replacement street, but cites the Pan-American Highway as the world's longest ``motorable road''. Is the “Yonge Street” the longest street in the world? Was the “Yonge Street” the longest street in the world in the past? Will the “Yonge Street” become the longest street in the world? Was Yonge Street the longest street in the world at some point? Was the 'Yonge Street' the longest street in the world before 2000? Was the 'Yonge Street' the …

Content truncated to 500 words