Domain-Aware Dependency Parsing for Questions

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4562–4568 August 1–6, 2021. ©2021 Association for Computational Linguistics 4562 Domain-Aware Dependency Parsing for Questions Aparna Garimella∗ Adobe Research garimell@adobe.com Laura Chiticariu IBM Research chiti@us.ibm.com Yunyao Li IBM Research yunyaoli@us.ibm.com Abstract Parsing natural language questions in specific domains is crucial to a wide range of applica- tions from question-answering to dialog sys- tems. Pre-trained parsers are usually trained on corpora dominated by non-questions, and thus perform poorly on domain-specific ques- tions. Retraining parsers with domain-specific questions labeled with syntactic parse trees is expensive, as these annotations require linguis- tic expertise. In this paper, we propose an automatic labeled domain question generation framework by leveraging domain knowledge and seed domain questions. We evaluate our approach in two domains, and release the gen- erated question datasets. Our experimental re- sults demonstrate that auto-generated labeled questions indeed lead to significant (4.9% − 9%) increase in the accuracy of state-of-the-art (SoTA) parsers on domain questions. 1 Introduction Understanding questions is the first step towards building accurate and reliable natural language in- terfaces. Recent works on Google Assistant, or IBM Watson focus on building domain-specific conversational agents. In this paper, we focus on syntactic parsing of domain-specific questions, which is crucial in domain-specific agents. The accuracy of syntactic parsers is known to depend on the syntactic similarities between the training data and application text. However, questions are often underrepresented in classic training corpora. In Penn TreeBank, only 0.5% of sentences from the Wall Street Journal are questions, with majority being rhetorical in nature, and those occurring in conversations (starting with interrogatives wh-/how, imperatives show me, name, or yes/no questions) are heavily underrepresented. Recognizing this ∗This work was done when the author was at IBM Re- search. problem, Judge et al. (2006) introduced Question- Bank, a labeled corpus of 4,000 general questions. However, domain-specific questions are often underrepresented in general purpose question cor- pora, leading to their poor parsing performances; e.g., in Show me Neil’s insider transactions since 2011, a SoTA parser (Nivre et al., 2016a) trained on Universal Dependencies (UD) English TreeBank (Silveira et al., 2014) and QuestionBank attaches since to show, instead of transactions, causing the system to misinterpret Neil’s insider transactions since 2011, to all his transactions. In Will it rain tomorrow by noon?, tomorrow is attached to rain with a wrong dependency relation (dobj instead of nmod:tmod), causing the system to miss the tempo- ral aspect of the question. A natural solution to obtain accurate domain- specific parsers is to train them on domain-specific corpora. However, obtaining domain-specific ques- tions is difficult. Moreover, annotating questions for parse trees is tedious, prone to errors and incon- sistencies, and requires linguistic expertise. Petrov et al. (2010) proposed uptraining, training a parser on the output of a slower, more accurate parser. For acceptable performance, the unlabeled corpus must be large (100,000 questions). Our method is applicable when such a large corpus is not avail- able. Inspired by (Wang et al., 2015) who showed that semantic …

Content truncated to 500 words