Articles

Previous studies have shown that initializ- ing neural machine translation (NMT) mod- els with the pre-trained language models (LM) can speed up the model training and boost the model performance. In this work, we identify a critical side-effect of pre-training for NMT, which is due to the discrepancy between the …...

The NLP community has seen substantial re- cent interest in grounding to facilitate inter- action between language technologies and the world. However, as a community, we use the term broadly to reference any linking of text to data or non-textual modality. In contrast, Cog- nitive Science more formally defines “ground- …...

Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Trans- formers is memory-intensive, which renders the search process infeasible. To this end, we propose a multi-split reversible network and combine it with DARTS. Specifically, we de- vise a backpropagation-with-reconstruction al- gorithm so that …...

Chinese word segmentation (CWS) is a fun- damental task for Chinese information pro- cessing, which always suffers from out-of- vocabulary word issues, especially when it is tested on data from different sources. Al- though one possible solution is to use more training data, in real applications, these data are stored …...

Automated event extraction in social science applications often requires corpus-level evalu- ations: for example, aggregating text predic- tions across metadata and unbiased estimates of recall. We combine corpus-level evalu- ation requirements with a real-world, social science setting and introduce the INDIAPO- LICEEVENTS corpus—all 21,391 sentences from 1,257 English-language Times of …...

We present a simplified, task-agnostic multi- modal pre-training approach that can accept either video or text input, or both for a va- riety of end tasks. Existing pre-training are task-specific by adopting either a single cross- modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or …...

Pre-trained language models (e.g., BERT) sig- nificantly alleviate two traditional challeng- ing problems for Chinese word segmentation (CWS): segmentation ambiguity and out-of- vocabulary (OOV) words. However, such im- provements are usually achieved on traditional benchmark datasets and not close to an im- portant goal of CWS: practicability (i.e., low complexity …...

Reporting and providing test sets for harm- ful bias in NLP applications is essential for building a robust understanding of the cur- rent problem. We present a new observation of gender bias in a downstream NLP appli- cation: marked attribute bias in natural lan- guage inference. Bias in downstream appli- …...

The drastic decrease in mobile SMS costs turned phone users more prone to spam mes- sages, usually with unwanted marketing or questionable content. As such, researchers have proposed different methods for detecting SMS spam messages. This paper presents a technique for embedding SMS messages into vector spaces that is suitable …...

Despite recent progress, learning new tasks through language instructions remains an ex- tremely challenging problem. On the AL- FRED benchmark for task learning, the pub- lished state-of-the-art system only achieves a task success rate of less than 10% in an un- seen environment, compared to the human per- formance of …...

Many commonsense reasoning NLP tasks in- volve choosing between one or more possi- ble answers to a question or prompt based on knowledge that is often implicit. Large pre- trained language models (PLMs) can achieve near-human performance on such tasks, while providing little human-interpretable evidence of the underlying reasoning they …...

Dataless text classification is capable of classi- fying documents into previously unseen labels by assigning a score to any document paired with a label description. While promising, it crucially relies on accurate descriptions of the label set for each downstream task. This re- liance causes dataless classifiers to be highly …...

Is it possible to use natural language to inter- vene in a model’s behavior and alter its pre- diction in a desired way? We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes. Specif- ically, we propose a new language under- …...

How much information do NLP tasks really need from a transformer’s attention mecha- nism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accu- racies. However, this …...

We introduce a language modeling architec- ture which operates over sequences of im- ages, or over multimodal sequences of images with associated labels. We use this architec- ture alongside other embedding models to in- vestigate a category of signs called complex graphemes (CGs) in the undeciphered proto- Elamite script. We …...

An attention matrix of a transformer self- attention sublayer can provably be decom- posed into two components and only one of them (effective attention) contributes to the model output. This leads us to ask whether visualizing effective attention gives different conclusions than interpretation of standard at- tention. Using a subset …...

Motivated by recent advancements in gram- matical error correction in English and exist- ing issues in the field, we describe a new re- source, an annotated learner corpus of Rus- sian, extracted from the Lang-8 language learn- ing website. This new dataset is benchmarked against two grammatical error correction mod- …...

Understanding who blames or supports whom in news text is a critical research question in computational social science. Traditional methods and datasets for sentiment analysis are, however, not suitable for the domain of political text as they do not consider the direc- tion of sentiments expressed between entities. In this …...

Multiple different responses are often plau- sible for a given open domain dialog con- text. Prior work has shown the importance of having multiple valid reference responses for meaningful and robust automated evaluations. In such cases, common practice has been to collect more human written references. How- ever, such collection …...

Text-to-Graph extraction aims to automati- cally extract information graphs consisting of mentions and types from natural language texts. Existing approaches, such as table filling and pairwise scoring, have shown impressive performance on various information extraction tasks, but they are difficult to scale to datasets with longer input texts because of …...