Bayesian Learning for Neural Dependency Parsing

Proceedings of NAACL-HLT 2019, pages 3509–3519 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3509 Bayesian Learning for Neural Dependency Parsing Ehsan Shareghi,♣ Yingzhen Li,♠ Yi Zhu,♣ Roi Reichart,♢ Anna Korhonen♣ ♣ Language Technology Lab, DTAL, University of Cambridge ♠Microsoft Research Cambridge, ♢Technion, IIT ♣{es776, yz568, alk23}@cam.ac.uk ♠Yingzhen.Li@microsoft.com, ♢roiri@technion.ac.il Abstract While neural dependency parsers provide state- of-the-art accuracy for several languages, they still rely on large amounts of costly labeled training data. We demonstrate that in the small data regime, where uncertainty around param- eter estimation and model prediction matters the most, Bayesian neural modeling is very ef- fective. In order to overcome the computa- tional and statistical costs of the approximate inference step in this framework, we utilize an efficient sampling procedure via stochastic gradient Langevin dynamics to generate sam- ples from the approximated posterior. More- over, we show that our Bayesian neural parser can be further improved when integrated into a multi-task parsing and POS tagging frame- work, designed to minimize task interference via an adversarial procedure. When trained and tested on 6 languages with less than 5푘 training instances, our parser consistently out- performs the strong BiLSTM baseline (Kiper- wasser and Goldberg, 2016). Compared with the BiAFFINE parser (Dozat et al., 2017) our model achieves an improvement of up to 3% for Vietnamese and Irish, while our multi-task model achieves an improvement of up to 9% across five languages: Farsi, Russian, Turkish, Vietnamese, and Irish. 1 Introduction Dependency parsing is essential for many Natural Language Processing (NLP) tasks (Levy and Gold- berg, 2014; Angeli et al., 2015; Toutanova et al., 2016; Hadiwinoto and Ng, 2017; Marcheggiani et al., 2017). While earlier work on dependency parsing required careful feature engineering (Mc- Donald et al., 2005b; Koo et al., 2008), this has become less of a concern in recent years with the emergence of deep neural networks (Kiperwasser and Goldberg, 2016; Dozat et al., 2017). Nonethe- less, an accurate parser still requires a large amount of labeled data for training, which is costly to ob- tain, while the lack of data often causes overfitting and poor generalization. Several approaches for parsing in the small data regime have been proposed. These include augmenting input data with pretrained embed- ding (Dozat et al., 2017; Che et al., 2018), lever- aging unannotated data via semi-supervised learn- ing (Corro and Titov, 2018), predicting based on a pool of high probability trees (Niculae et al., 2018; Keith et al., 2018), and transferring annotation or model across languages (Agic et al., 2016; Lacroix et al., 2016; Rasooli and Collins, 2017). Despite the empirical success of these approaches, an inherent problem still holds: The maximum likelihood pa- rameter estimation (MLE) in deep neural networks (DNNs) introduces statistical challenges at both estimation (training), due to the risk of overfitting, and at test time as the model ignores the uncertainty around the estimated parameters. When training data is small these challenges are more pronounced. The Bayesian paradigm provides a statistical framework which addresses …

Content truncated to 500 words