Human-in-the-Loop Parsing

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2337–2342, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Human-in-the-Loop Parsing Luheng He Julian Michael Mike Lewis Luke Zettlemoyer Computer Science & Engineering University of Washington Seattle, WA {luheng,julianjm,mlewis,lsz}@cs.washington.edu Abstract This paper demonstrates that it is possible for a parser to improve its performance with a hu- man in the loop, by posing simple questions to non-experts. For example, given the first sentence of this abstract, if the parser is un- certain about the subject of the verb “pose,” it could generate the question What would pose something? with candidate answers this pa- per and a parser. Any fluent speaker can answer this question, and the correct answer resolves the original uncertainty. We apply the approach to a CCG parser, converting un- certain attachment decisions into natural lan- guage questions about the arguments of verbs. Experiments show that crowd workers can an- swer these questions quickly, accurately and cheaply. Our human-in-the-loop parser im- proves on the state of the art with less than 2 questions per sentence on average, with a gain of 1.7 F1 on the 10% of sentences whose parses are changed. 1 Introduction The size of labelled datasets has long been recog- nized as a bottleneck in the performance of nat- ural language processing systems (Marcus et al., 1993; Petrov and McDonald, 2012). Such datasets are expensive to create, requiring expert linguists and extensive annotation guidelines. Even rela- tively large datasets, such as the Penn Treebank, are much smaller than required—as demonstrated by improvements from semi-supervised learning (Søgaard and Rishøj, 2010; Weiss et al., 2015). We take a step towards cheap, reliable annotations by introducing human-in-the-loop parsing, where Temple also said Sea Containers’ plan raises numer- ous legal, regulatory, financial and fairness issues, but didn’t elaborate. Q: What didn’t elaborate? [1] **** Temple [2] * Sea Containers’ plan [3] None of the above. Table 1: An automatically generated query from CCGbank. 4 out of 5 annotators correctly answered Temple, providing a signal that can be used to improve parse predictions. non-experts improve parsing accuracy by answering questions automatically generated from the parser’s output. We develop the approach for CCG parsing, leveraging the link between CCG syntax and seman- tics to convert uncertain attachment decisions into natural language questions. The answers are used as soft constraints when re-parsing the sentence. Previous work used crowdsourcing for less struc- tured tasks such as named entity recognition (Wer- ling et al., 2015) and prepositional phrase attach- ment (Jha et al., 2010). Our work is most related to that of Duan et al. (2016), which automatically generates paraphrases from n-best parses and gained significant improvement by re-training from crowd- sourced judgments on two out-of-domain datasets. Choe and McClosky (2015) improve a parser by cre- ating paraphrases of sentences, and then parsing the sentence and its paraphrase jointly. Instead of using paraphrases, we build on the approach of QA-SRL (He et al., 2015), which shows that untrained crowd …

Content truncated to 500 words