Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4362–4379 June 6–11, 2021. ©2021 Association for Computational Linguistics 4362 SWORDS : A Benchmark for Lexical Substitution with Improved Data Coverage and Quality Mina Lee* Chris Donahue* Robin Jia Stanford University {minalee, cdonahue, robinjia, aiyabor, pliang}@cs.stanford.edu Alexander Iyabor Percy Liang Abstract We release a new benchmark for lexical sub- stitution, the task of finding appropriate substi- tutes for a target word in a context. For writing, lexical substitution systems can assist humans by suggesting words that humans cannot eas- ily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to hu- mans. Furthermore, annotators often provide substitutes of low quality, which are not actu- ally appropriate in the given context. We col- lect higher-coverage and higher-quality data by framing lexical substitution as a classifica- tion problem, guided by the intuition that it is easier for humans to judge the appropriate- ness of candidate substitutes than conjure them from memory. To this end, we use a context- free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our SWORDS benchmark has 3x as many substitutes per target word for the same level of quality, and its substitutes are 1.4x more appropriate (based on human judge- ment) for the same number of substitutes. 1 Introduction Imagine you are writing the message “I read an amazing paper today” to a colleague, but you want to choose a more descriptive adjective to replace “amazing.” At first you might think of substitutes like “awesome” and “great,” but feel that these are also unsatisfactory. You turn to a thesaurus for inspiration, but among reasonable alternatives like “incredible” and “fascinating” are words like “prodigious” which do not quite fit in your context. Ultimately, you choose to go with “fascinating,” but reaching this decision required a non-trivial amount of time and effort. *Equal contribution. Research on lexical substitution (McCarthy, 2002; McCarthy and Navigli, 2007; Erk and Padó, 2008; Szarvas et al., 2013; Kremer et al., 2014; Melamud et al., 2015; Hintz and Biemann, 2016; Zhou et al., 2019; Arefyev et al., 2020) considers the task of replacing a target word in context with appropriate substitutes. There are two widely-used English benchmarks for this task: SEMEVAL (McCarthy and Navigli, 2007) and COINCO (Kremer et al., 2014). For both benchmarks, data was collected by asking human annotators to think of substitutes from memory. Because lexical substitution was originally proposed as a means for evaluating word sense disambiguation systems (McCarthy, 2002), this data collection strategy was designed to avoid a bias towards any particular word sense inventory. In this work, we consider a different use case for lexical substitution: writing assistance. For this use case, we are interested in evaluating a system’s ability to produce appropriate substitutes that are likely to be difficult …

Content truncated to 500 words