Stay Together: A System for Single and Split-antecedent Anaphora Resolution

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 352–365 June 6–11, 2021. ©2021 Association for Computational Linguistics 352 Fool Me Twice: Entailment from Wikipedia Gamification Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger Google Research { eisenjulian, bdhingra, jbulian, bboerschinger} @google.com Jordan Boyd-Graber∗ CS, iSchool, UMIACS, LSC University of Maryland jbg@umiacs.umd.edu Abstract We release FOOLMETWICE (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial exam- ples, drastically lowering the number of exam- ples that can be solved using “shortcuts” com- pared to other entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The sec- ond one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players “pay” to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play be- tween motivated players leads to diverse strate- gies for crafting claims, such as temporal infer- ence and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and game code.1 1 Introducing a Game of Challenging Claims Given a statement—and a large collection of tex- tual knowledge—how do you find evidence that shows a reader that the statement is true or false? This problem takes on multiple forms in the natu- ral language processing (NLP) community. Given only a single statement and a single sentence, this decision process is called recognizing textual entail- ment (Dagan et al., 2010, RTE) or natural language inference (Bowman et al., 2015; Williams et al., 2018, NLI). Given a single statement and a vast pool of possible evidence (e.g., all of Wikipedia), this problem is called verification (Thorne et al., 2018; Jiang et al., 2020). ∗Work completed while a Visiting Research Scientist at Google. 1https://github.com/google-research/ fool-me-twice Stage 1: Players write claims to fool others "Venus's craters are difficult to measure due to erosion." Stage 2: Players mark evidence that entails/refutes the claim On Earth it is caused by wind and rain erosion. On Venus, about 85% of the craters are in pristine condition. Venusian craters range from 3 to 280 km. Stage 3: Players spot the refuted claim "Snoop Dogg portrayed Moses in a rap battle" "Venus's craters are difficult to measure due to erosion." Stage 4: Players get points Players that correctly spotted the refuted claim and authors of challenging claims are rewarded 100 Stage 5: Automatically build dataset Keep highest quality claims with their selected evidence FM2 Repeat Figure 1: Overview of the data generation pipeline. In stages 1 to 4, players write challenging claims either entailed or refuted by evidence from Wikipedia (Sec- tion 3.1). They are then tasked to spot the refuted claim among a …

Content truncated to 500 words