Self Supervised Training
Related Papers
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale
AbstractWe study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate …...
A Bidirectional Transformer Based Alignment Model for Unsupervised Word Alignment
AbstractWord alignment and machine translation are two closely related tasks. Neural translation models, such as RNN-based and Transformer models, employ a target-to-source attention mechanism which can provide rough word alignments, …...
Changing the Mind of Transformers for Topically-Controllable Language Generation
AbstractLarge Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation …...
Dict-BERT: Enhancing Language Model Pre-training with Dictionary
AbstractPre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word …...
Dilated Convolutional Neural Networks for Lightweight Diacritics Restoration
AbstractDiacritics restoration has become a ubiquitous task in the Latin-alphabet-based English-dominated Internet language environment. In this paper, we describe a small footprint 1D dilated convolution-based approach which operates on a …...
TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method
AbstractLyric-to-melody generation is an important task in automatic songwriting. Previous lyric-to-melody generation systems usually adopt end-to-end models that directly generate melodies from lyrics, which suffer from several issues: 1) lack …...