Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643–653, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language Luheng He Mike Lewis Luke Zettlemoyer Computer Science & Engineering University of Washington Seattle, WA {luheng,mlewis,lsz}@cs.washington.edu Abstract This paper introduces the task of question- answer driven semantic role labeling (QA-SRL), where question-answer pairs are used to represent predicate-argument structure. For example, the verb “intro- duce” in the previous sentence would be labeled with the questions “What is in- troduced?”, and “What introduces some- thing?”, each paired with the phrase from the sentence that gives the correct answer. Posing the problem this way allows the questions themselves to define the set of possible roles, without the need for prede- fined frame or thematic role ontologies. It also allows for scalable data collection by annotators with very little training and no linguistic expertise. We gather data in two domains, newswire text and Wikipedia articles, and introduce simple classifier- based models for predicting which ques- tions to ask and what their answers should be. Our results show that non-expert anno- tators can produce high quality QA-SRL data, and also establish baseline perfor- mance levels for future work on this task. 1 Introduction Semantic role labeling (SRL) is the widely stud- ied challenge of recovering predicate-argument structure for natural language words, typically verbs. The goal is to determine “who does what to whom,” “when,” and “where,” etc. (Palmer et al., 2010; Johansson and Nugues, 2008). However, this intuition is difficult to formalize and funda- mental aspects of the task vary across efforts, for example FrameNet (Baker et al., 1998) models a large set of interpretable thematic roles (AGENT, PATIENT, etc.) while PropBank (Palmer et al., 2005) uses a small set of verb-specific roles UCD finished the 2006 championship as Dublin champions , by beating St Vincents in the final . finished beating Who finished something? - UCD What did someone finish? - the 2006 championship What did someone finish something as? - Dublin champions How did someone finish something? - by beating St Vincents in the final Who beat someone? - UCD When did someone beat someone? - in the final Who did someone beat? - St Vincents Figure 1: QA-SRL annotations for a Wikipedia sentence. (ARG0, ARG1, etc.). Existing task definitions can be complex and require significant linguistic ex- pertise to understand,1 causing challenges for data annotation and use in many target applications. In this paper, we introduce a new question- answer driven SRL task formulation (QA-SRL), which uses question-answer pairs to label verbal predicate-argument structure. For example, for the sentence in Figure 1, we can ask a short ques- tion containing a verb, e.g. “Who finished some- thing?”, and whose answer is a phrase from the original sentence, in this case “UCD.” The answer tells us that “UCD” is an argument of “finished,” while the question provides an indirect label on the role that “UCD” plays. …

Content truncated to 500 words