Controlled Crowdsourcing for High-Quality QA-SRL Annotation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008–7013 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7008 Controlled Crowdsourcing for High-Quality QA-SRL Annotation Paul Roit1, Ayal Klein1, Daniela Stepanov1, Jonathan Mamou3, Julian Michael2, Gabriel Stanovsky2,4 , Luke Zettlemoyer2,5 , and Ido Dagan1 1Department of Computer Science, Bar-Ilan University, Ramat-Gan, Israel 2Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, USA 3Intel AI Lab, Israel 4Allen Institute for AI, Seattle, USA 5Facebook AI Research {plroit, ayal.s.klein, daniela.stepanov}@gmail.com jonathan.mamou@intel.com {julianjm, gabis, lsz}@cs.washington.edu dagan@cs.biu.ac.il Abstract Question-answer driven Semantic Role Label- ing (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially at- tainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were released. Trying to replicate the QA-SRL annotation for new texts, we found that the resulting annotations were lacking in quality, particularly in coverage, making them insufficient for further research and evaluation. In this paper, we present an improved crowd- sourcing protocol for complex semantic anno- tation, involving worker selection and train- ing, and a data consolidation phase. Apply- ing this protocol to QA-SRL yielded high- quality annotation with drastically higher cov- erage, producing a new gold evaluation dataset. We believe that our annotation protocol and gold standard will facilitate future replicable research of natural semantic annotations. 1 Introduction Semantic Role Labeling (SRL) provides explicit annotation of predicate-argument relations. Com- mon SRL schemes, particularly PropBank (Palmer et al., 2005) and FrameNet (Baker et al., 1998), rely on predefined role inventories and extensive predicate lexicons. Consequently, SRL annotation of new texts requires substantial efforts involving expert annotation, and possibly lexicon extension, limiting scalability. Aiming to address these limitations, Question- Answer driven Semantic Role Labeling (QA-SRL) (He et al., 2015) labels each predicate-argument relationship with a question-answer pair, where nat- ural language questions represent semantic roles, and answers correspond to arguments (see Table 1). This approach follows the colloquial perception of semantic roles as answering questions about the predicate (“Who did What to Whom, When, Where and How”, with, e.g., “Who” corresponding to the agent role). QA-SRL carries two attractive promises. First, using a question-answer format makes the annota- tion task intuitive and easily attainable by laymen, as it does not depend on linguistic resources (e.g. role lexicons), thus facilitating greater annotation scalability. Second, by relying on intuitive human comprehension, these annotations elicit a richer argument set, including valuable implicit seman- tic arguments not manifested in syntactic structure (highlighted in Table 1). The importance of im- plicit arguments has been recognized in the litera- ture (Cheng and Erk, 2018; Do et al., 2017; Gerber and Chai, 2012), yet they are mostly overlooked by common SRL formalisms and tools. Overall, QA-SRL largely subsumes predicate- argument information captured by traditional SRL schemes, which were shown beneficial for complex downstream tasks, such as dialog modeling (Chen et al., 2013), machine comprehension (Wang et al., 2015) and cross-document coreference (Barhom et al., 2019). At the same time, it contains …

Content truncated to 500 words