Limitations in learning an interpreted language with recurrent models

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 384–386 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 384 Limitations in learning an interpreted language with recurrent models Denis Paperno Loria, Centre national de la recherche scientifique (CNRS) Campus Scientifique, BP 239, 54506 Vandoeuvre-l`es-Nancy, France paperno@ucla.edu Abstract In this submission I report work in progress on learning simplified interpreted languages by means of recurrent models. The data is constructed to reflect core properties of nat- ural language as modeled in formal syntax and semantics. Preliminary results suggest that LSTM networks do generalise to compo- sitional interpretation, albeit only in the most favorable learning setting. Motivation. Despite showing impressive per- formance on certain tasks, neural networks are still far from showing natural language understanding at a human level, cf. Paperno et al. (2016). In a sense, it is not even clear what kind of neural ar- chitecture is capable of learning natural language semantics in all its complexity, with recurrent and convolutional models being currently tried on var- ious tasks. One can hope to make progress towards the challenging goal of natural language understand- ing by taking into account what is known about language structure and language processing in hu- mans. With this in mind, it is possible to formulate certain preliminary desiderata for an adequate nat- ural language understanding model. First, language processing in humans is known to be sequential; people process and interpret lin- guistic input on the fly, without any lookahead and without waiting for the linguistic structure to be completed. This property, which has serious po- tential consequences for the cognitive architecture (Christiansen and Chater, 2016), gives a certain degree of cognitive plausibility to unidirectional recurrent models compared to other neural archi- tectures, at last in their current implementations. Second, natural language can exploit recursive structures: natural language syntax consists of constructions, represented in formal grammars as rewrite rules, which can recursively embed other constructions of the same kind. For example, noun phrases can in principle consist of a single proper noun (e.g. Ann) but can also, among other pos- sibilities, be built from other noun phrases recur- sively via the possessive construction, as in Ann’s child, Ann’s child’s friend, Ann’s child’s friend’s parent etc. The possessive construction can be de- scribed by the rewrite rule NP −→ NP’s N. Third, the recursive syntactic structure drives compositional semantic interpretation. The mean- ing of the noun phrase Ann’s child’s friend is not merely the sum of the meanings of the individual words (in which case it would have been semanti- cally equivalent to Ann’s friend’s child). Rather, to interpret a complex expression correctly, one has to follow the syntactic structure, first identify- ing the meaning of the smaller constituent (Ann’s friend), and then computing the meaning of the whole on its basis. Fourth, semantic compositionality can be for- malized as function application, with one con- stituent in a complex structure corresponding to an argument of a function that another constituent …

Content truncated to 500 words