Exploring Looping Effects in RNN-based Architectures

Exploring Looping Effects in RNN-based Architectures Andrei Shcherbakovµ Saliha Murado˘gluΩΦ Ekaterina Vylomovaµ ΩThe Australian National University (ANU) µThe University of Melbourne ΦARC Centre of Excellence for the Dynamics of Language (CoEDL) sandreas@unimelb.edu.au, saliha.muradgolu@anu.edu.au ekaterina.vylomova@unimelb.edu.au Abstract The paper investigates repetitive loops, a com- mon problem in contemporary text generation (such as machine translation, language mod- elling, morphological inflection) systems. We hypothesized that a model’s failure to distin- guish respective latent states for different po- sitions in an output sequence may be the pri- mary cause of the looping. Therefore, we pro- pose adding a position-aware discriminating factor to the model in attempt to reduce that effect. We conduct a study on neural models with recurrent units by explicitly altering their decoder internal state. We use a task of mor- phological reinflection as a proxy to study the effects of the changes. Our results show that the probability of the occurrence of repetitive loops is significantly reduced by introduction of an extra neural decoder output. The out- put should be specifically trained to produce gradually increasing value upon generation of each character of a given sequence. We also explored variations of the technique and found that feeding the extra output back to the de- coder amplifies the positive effects. 1 Introduction Over the last few years we witnessed a significant progress in the field of Natural Language Process- ing (NLP). Many state-of-the-art models are based on neural architectures with recurrent units. For instance, Sutskever et al. (2014) proposed one of the first neural machine translation models that achieved results comparable with statistical mod- els. Similarly, Plank et al. (2016) introduced a neural POS tagging model as a new state-of-the-art on the task. Recently, neural architectures almost superseded non-neural (finite-state or rule-based) approaches in morphology modelling tasks such as morphological reinflection (Cotterell et al., 2016, 2017) with average accuracy being over 90% on high-resource languages. Error analysis conducted by Gorman et al. (2019) demonstrated that among general misprediction errors such as syncretism, the models also produce certain “silly” errors that human learners do not make. One case of such errors, a looping error, is particularly notable. This type of error is not specific to the task and several other papers reported a similar problem (Holtzman et al., 2019; Vakilipourtakalou and Mou, 2020). Still, the causes and the nature of the error remains under-studied. Here we provide some insights on the causes of the issues and possible remedy to it. We consider morphological reinflection task for our experiments since it has low time and space requirements and, therefore, allows us to reproduce cases of looping in sufficient quantities and analyse them relatively easy. 2 Morphological reinflection task Morphological inflection is the task of generating a target word form (e.g., “runs”) from its lemma (“to run”) and a set of target morphosyntactic features (tags, “Verb;Present Tense;Singular;3rd Person”). The task is called morphological reinflection when the lemma form is replaced with any other form and, optionally, its morphosyntactic features. This is a type of string-to-string transduction …

Content truncated to 500 words