LanguageModelling
Related Papers
Trigger-Pair Predictors in Parsing and Tagging
In this article, we apply to natural language parsing and tagging the device of trigger- pair predictors, previously employed exclu- sively within the field of language mod- elling for speech …...
The RODRIGO Database
AbstractAnnotation of digitized pages from historical document collections is very important to research on automatic extraction of text blocks, lines, and handwriting recognition. We have recently introduced a new handwritten …...
Fast Gated Neural Domain Adaptation: Language Model as a Case Study
AbstractNeural network training has been shown to be advantageous in many natural language processing applications, such as language modelling or machine translation. In this paper, we describe in detail a …...
Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models
AbstractResearchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplexity on sequence learning tasks such …...
Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation
AbstractThe Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens …...
No Data to Crawl? Monolingual Corpus Creation from PDF Files of Truly low-Resource Languages in Peru
AbstractWe introduce new monolingual corpora for four indigenous and endangered languages from Peru: Shipibo-konibo, Ashaninka, Yanesha and Yine. Given the total absence of these languages in the web, the extraction …...
Re-framing Incremental Deep Language Models for Dialogue Processing with Multi-task Learning
AbstractWe present a multi-task learning framework to enable the training of one universal incremental dialogue processing model with four tasks of disfluency detection, language modelling, part-of-speech tagging and utterance segmentation …...
Language Modelling as a Multi-Task Problem
AbstractIn this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic …...
Reservoir Transformers
AbstractWe demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore …...
Improving Code-switched ASR with Linguistic Information
AbstractThis paper seeks to improve the performance of automatic speech recognition (ASR) systems operating on code-switched speech. Code-switching refers to the alternation of languages within a conversation, a phenomenon that …...
Revisiting Syllables in Language Modelling and Their Application on Low-Resource Machine Translation
AbstractLanguage modelling and machine translation tasks mostly use subword or character inputs, but syllables are seldom used. Syllables provide shorter sequences than characters, require less-specialised extracting rules than morphemes, and …...
State Spaces Aren’t Enough: Machine Translation Needs Attention
AbstractStructured State Spaces for Sequences (S4) is a recently proposed sequence model with successful applications in various tasks, e.g. vision, language modelling, and audio. Thanks to its mathematical formulation, it …...