Articles

Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of cross- lingual transfer tasks. However, it remains unknown whether the optimization for differ- ent languages conditions the capacity of the models to generalize over syntactic structures, and …...

Context guides comprehenders’ expectations during language processing, and information- theoretic surprisal is commonly used as an in- dex of cognitive processing effort. However, prior work using surprisal has considered only within-sentence context, using n-grams, neural language models, or syntactic structure as con- ditioning context. In this paper, we extend the …...

Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated re- sources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert an- notated monolingual datasets when …...

We introduce a new approach for the task of Controllable Text Simplification, where sys- tems rewrite a complex English sentence so that it can be understood by readers at differ- ent grade levels in the US K-12 system. It uses a non-autoregressive model to iteratively edit an input sequence and …...

Texts convey sophisticated knowledge. How- ever, texts also convey sensitive information. Despite the success of general-purpose lan- guage models and domain-specific mecha- nisms with differential privacy (DP), existing text sanitization mechanisms still provide low utility, as cursed by the high-dimensional text representation. The companion issue of uti- lizing sanitized texts …...

The recent advancement of pre-trained Trans- former models has propelled the development of effective text mining models across various biomedical tasks. However, these models are primarily learned on the textual data and often lack the domain knowledge of the entities to capture the context beyond the sentence. In this study, …...

With the ever-increasing complexity of neu- ral language models, practitioners have turned to methods for understanding the predictions of these models. One of the most well- adopted approaches for model interpretability is feature-based interpretability, i.e., ranking the features in terms of their impact on model predictions. Several prior studies have …...

The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approxi- mated by each word’s sample frequency in the corpus. This approach, being highly depen- dent on sample size, assigns zero probability …...

We propose a new task and dataset for a common problem in social science research: “upsampling” coarse document labels to fine- grained labels or spans. We pose the problem in a question answering format, with the an- swers providing the fine-grained labels. We provide a benchmark dataset and baselines on …...

Recent years have seen a rise in interest for cross-lingual transfer between languages with similar typology, and between languages of various scripts. However, the interplay be- tween language similarity and difference in script on cross-lingual transfer is a less stud- ied problem. We explore this interplay on cross-lingual transfer for …...

Multi-modal machine translation (MMT) aimed at using images to help disambiguate the target during translation and improving robustness, but some recent works showed that the contribution of visual features is either negligible or incremental. In this paper, we show that incorporating pre-trained (vision) language model (VLP) on the source side …...

Inspired by evidence that pretrained language models (LMs) encode commonsense knowl- edge, recent work has applied LMs to auto- matically populate commonsense knowledge graphs (CKGs). However, there is a lack of understanding on their generalization to mul- tiple CKGs, unseen relations, and novel enti- ties. This paper analyzes the ability …...

Online political advertising is a central aspect of modern election campaigning for influenc- ing public opinion. Computational analysis of political ads is of utmost importance in polit- ical science to understand the characteristics of digital campaigning. It is also important in computational linguistics to study features of political discourse and …...

Sentiment lexicons are instrumental for senti- ment analysis. One can use a set of sentiment words provided in a sentiment lexicon and a lexicon-based classifier to perform sentiment analysis. One major issue with this approach is that many sentiment words (from the lexicon) are domain dependent. That is, they may …...

Fixed length summarization aims at generat- ing summaries with a preset number of words or characters. Most recent researches incor- porate length information with word embed- dings as the input to the recurrent decoding unit, causing a compromise between length controllability and summary quality. In this work, we present an …...

Transformers have been shown to emulate log- ical deduction over natural language theories (logical rules expressed in natural language), reliably assigning true/false labels to candi- date implications. However, their ability to generate implications of a theory has not yet been demonstrated, and methods for recon- structing proofs of answers are …...

Automated Term Extraction (ATE), even though well-investigated, continues to be a challenging task. Approaches conventionally extract terms on corpus or document level and the benefits of neural models still remain un- derexplored with very few exceptions. We in- troduce three transformer-based term extrac- tion models operating on sentence level: a …...

In this paper, we analyze the interplay be- tween the use of offensive language and men- tal health. We acquired publicly available datasets created for offensive language identi- fication and depression detection and we train computational models to compare the use of offensive language in social media posts writ- ten …...

Natural language processing (NLP) methods for analyzing legal text offer legal scholars and practitioners a range of tools allowing to empirically analyze law on a large scale. However, researchers seem to struggle when it comes to identifying ethical limits to using NLP systems for acquiring genuine insights both about the …...

State-of-the-art Variational Auto-Encoders (VAEs) for learning disentangled latent representations give impressive results in discovering features like pitch, pause duration, and accent in speech data, leading to highly controllable text-to-speech (TTS) synthesis. However, these LSTM-based VAEs fail to learn latent clusters of speaker attributes when trained on limited or noisy datasets. …...