Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1150–1160, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1150 Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal Benoˆıt Crabb´e LLF - CNRS University of Paris - IUF Place Paul Ricoeur 75013 Paris France benoit.crabbe@ linguist.univ-paris-diderot.fr Murielle Popa-Fabre ALMANACH - INRIA - LLF University of Paris 2 rue Simone Iff 75012 Paris France murielle.fabre@inria.fr Christophe Pallier Cognitive Neuroimaging Lab INSERM-CEA Neurospin bat.145 Gif-sur-Yvette France christophe@pallier.org Abstract This paper describes a method of variable beam size inference for Recurrent Neural Net- work Grammar (RNNG) by drawing inspi- ration from sequential Monte-Carlo methods such as particle filtering. The paper studies the relevance of such meth- ods for speeding up the computations of di- rect generative parsing for RNNG. But it also studies the potential cognitive interpretation of the underlying representations built by the search method (beam activity) through anal- ysis of neuro-imaging signal through corre- lations with the neuro-imaging signal during naturalistic text listening 1 Introduction The paper provides a preliminary investigation of how the improvements in natural language parsing techniques can be used for modelling brain activ- ity during online sentence comprehension. Specif- ically, we focus on the RNNG generative parsing framework (Dyer et al., 2016), that allows us to ex- tract information-theoretic measures that are rele- vant for cognitive studies such as surprisal or en- tropy (Hale, 2016). RNNG is essentially a generative parsing frame- work but it is not straightforward to design a one- step generative parser. Dyer et al. (2016) achieves generative parsing with RNNG in two steps: by re- ranking a discriminative parser. Stern et al. (2017) points out that the main difficulty comes from lex- ical biases using naive beam search. They pro- pose new methods for modifying a classical word- synchronized beam search for avoiding the lexical bias problem. The paper presents a substantially different search method for parsing in one-step with a neu- ral generative parser, proposing a method inspired by sequential Monte-Carlo sampling and particle filtering (Doucet and Johansen, 2011; Buys and Blunsom, 2015). Crucially, the proposed method is naturally not sensitive to problems of lexical biases faced by standard beam search. As a result, the parser is framed as generative and strictly incremental (without lookahead) while remaining quite accu- rate and generally more efficient than a more tra- ditional beam search method. Not only, the pro- posed search method is a variable-beam search method, but as such, it naturally provides an addi- tional method to measure the activity required to make the parser progress incrementally and word- by-word that can, in principle, be used for cogni- tive modelling purposes. Taking advantage of our search method proper- ties, we introduce new parsing complexity mea- sures such as beam-size and beam-activity, adding to the well known entropy or surprisal that are already provided by the generative model. Fi- nally, we …

Content truncated to 500 words