Cross-Domain Generalization of Neural Constituency Parsers

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 323–330 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 323 Cross-Domain Generalization of Neural Constituency Parsers Daniel Fried∗ Nikita Kitaev∗ Computer Science DivisionDan Klein University of California, Berkeley {dfried,kitaev,klein}@cs.berkeley.edu Abstract Neural parsers obtain state-of-the-art results on benchmark treebanks for constituency parsing—but to what degree do they general- ize to other domains? We present three re- sults about the generalization of neural parsers in a zero-shot setting: training on trees from one corpus and evaluating on out-of-domain corpora. First, neural and non-neural parsers generalize comparably to new domains. Sec- ond, incorporating pre-trained encoder repre- sentations into neural parsers substantially im- proves their performance across all domains, but does not give a larger relative improvement for out-of-domain treebanks. Finally, despite the rich input representations they learn, neu- ral parsers still benefit from structured output prediction of output trees, yielding higher ex- act match accuracy and stronger generalization both to larger text spans and to out-of-domain corpora. We analyze generalization on English and Chinese corpora, and in the process obtain state-of-the-art parsing results for the Brown, Genia, and English Web treebanks. 1 Introduction Neural constituency parsers have obtained increas- ingly high performance when measured by F1 scores on in-domain benchmarks, such as the Wall Street Journal (WSJ) (Marcus et al., 1993) and Penn Chinese Treebank (CTB) (Xue et al., 2005). However, in order to construct systems useful for cross-domain NLP, we seek parsers that general- ize well to domains other than the ones they were trained on. While classical, non-neural parsers are known to perform better in their training do- mains than on out-of-domain corpora, their out-of- domain performance degrades in well-understood ways (Gildea, 2001; Petrov and Klein, 2007), and improvements in performance on in-domain ∗Equal contribution. treebanks still transfer to out-of-domain improve- ments (McClosky et al., 2006). Is the success of neural constituency parsers (Henderson 2004; Vinyals et al. 2015; Dyer et al. 2016; Cross and Huang 2016; Choe and Charniak 2016; Stern et al. 2017; Liu and Zhang 2017; Ki- taev and Klein 2018, inter alia) similarly transfer- able to out-of-domain treebanks? In this work, we focus on zero-shot generalization: training parsers on a single treebank (e.g. WSJ) and evaluating on a range of broad-coverage, out-of-domain tree- banks (e.g. Brown (Francis and Kuˇcera, 1979), Genia (Tateisi et al., 2005), the English Web Tree- bank (Petrov and McDonald, 2012)). We ask three questions about zero-shot generalization proper- ties of state-of-the-art neural constituency parsers: First, do non-neural parsers have better out-of- domain generalization than neural parsers? We might expect neural systems to generalize poorly because they are highly-parameterized, and may overfit to their training domain. We find that neural and non-neural parsers generalize sim- ilarly, and, encouragingly, improvements on in- domain treebanks still transfer to out-of-domain. Second, does pre-training particularly improve out-of-domain performance, or does it just gener- ally improve test accuracies? Neural parsers in- corporate rich representations of language that can easily be pre-trained …

Content truncated to 500 words