Articles

Understanding the political perspective shap- ing the way events are discussed in the media is increasingly important due to the dramatic change in news distribution. With the advance in text classification models, the performance of political perspective detection is also im- proving rapidly. However, current deep learn- ing based text …...

ELECTRA (Clark et al., 2020a) pretrains a dis- criminator to detect replaced tokens, where the replacements are sampled from a gener- ator trained with masked language modeling. Despite the compelling performance, ELEC- TRA suffers from the following two issues. First, there is no direct feedback loop from dis- criminator to …...

Pre-trained language models have been widely used in response generation for open-domain dialogue. These approaches are built within 4 frameworks: Transformer-ED, Transformer- Dec, Transformer-MLM and Transformer-AR. In this study, we experimentally compare them using both large and small-scale data. This re- veals that decoder-only architecture is better than stacked encoder-decoder, …...

Developing effective distributed representa- tions of source code is fundamental yet chal- lenging for many software engineering tasks such as code clone detection, code search, code translation and transformation. However, current code embedding approaches that rep- resent the semantic and syntax of code in a mixed way are less interpretable …...

Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Auto- mated visual storytelling (VIST) models, how- ever, make poor use of external knowledge and iterative generation …...

Pre-trained language models (PTLMs) ac- quire domain-independent linguistic knowl- edge through pre-training with massive textual resources. Additional pre-training is effective in adapting PTLMs to domains that are not well covered by the pre-training corpora. Here, we focus on the static word embeddings of PTLMs for domain adaptation to teach PTLMs …...

Visual dialogue is a challenging task since it needs to answer a series of coherent ques- tions on the basis of understanding the vi- sual environment. Previous studies focus on the implicit exploration of multimodal co- reference by implicitly attending to spatial image features or object-level image features but neglect …...

Text generation from semantic parses is to generate textual descriptions for formal rep- resentation inputs such as logic forms and SQL queries. This is challenging due to two reasons: (1) the complex and intensive in- ner logic with the data scarcity constraint, (2) the lack of automatic evaluation metrics for …...

Pre-trained language models (PLMs) achieve great success in NLP. However, their huge model sizes hinder their applications in many practical systems. Knowledge distillation is a popular technique to compress PLMs, which learns a small student model from a large teacher PLM. However, the knowledge learned from a single teacher may …...

Argumentation exposes individuals to conflict- ing viewpoints and can help them make more informed decisions based on the pros and cons of a particular issue. While recent studies of ar- gumentation in Natural Language Processing have mainly focused on understanding the ef- fect of various factors of persuasion (i.e. the …...

While non-autoregressive (NAR) models are showing great promise for machine translation (MT), their use is limited by their dependence on knowledge distillation from au- toregressive models. To address this issue, we seek to understand why distillation is so effec- tive. Prior work suggests that distilled training data is less complex …...

Code generation is the task of generating code snippets from input user specifications in nat- ural language. Leveraging the linguistically- motivated hierarchical structure of the input can benefit code generation, especially since the specifications are complex sentences con- taining multiple variables and operations over various data structures. Moreover, recent advances …...

A growing number of people engage in online health forums, making it important to under- stand the quality of the advice they receive. In this paper, we explore the role of expertise in responses provided to help-seeking posts regarding mental health. We study the dif- ferences between (1) interactions with …...

Neural topic modeling approach has been at- tracting much attention recently as it is able to leverage the advantages of both neural net- works and probabilistic topic models. Previ- ous works have proposed several models that are based on this framework and obtained im- pressive experimental results compared to tra- …...

As a language model that integrates traditional symbolic operations and flexible neural repre- sentations, recurrent neural network grammars (RNNGs) have attracted great attention from both scientific and engineering perspectives. However, RNNGs are known to be harder to scale due to the difficulty of batched training. In this paper, we propose …...

Contextualized word representations have proven useful for various natural language pro- cessing tasks. However, it remains unclear to what extent these representations can cover hand-coded semantic information such as se- mantic frames, which specify the semantic role of the arguments associated with a pred- icate. In this paper, we focus …...

Generating context-aware language that em- bodies diverse emotions is an important step towards building empathetic NLP systems. In this paper, we propose a formulation of mod- ulated layer normalization—a technique in- spired by computer vision—that allows us to use large-scale language models for emo- tional response generation. In automatic and …...

Knowledge Bases (KBs) are easy to query, ve- rifiable, and interpretable. They however scale with man-hours and high-quality data. Masked Language Models (MLMs), such as BERT, scale with computing power as well as unstruc- tured raw text data. The knowledge contained within these models is however not directly in- terpretable. …...

Emphasizing words in presentation slides al- lows viewers to direct their gaze to focal points without reading the entire slide, retaining their attention on the speaker. Despite many stud- ies on automatic slide generation, few have ad- dressed helping authors choose which words to emphasize. Motivated by this, we study …...

Recent advancements in transformer-based models have greatly improved the ability of Question Answering (QA) systems to provide correct answers; in particular, answer sentence selection (AS2) models, core components of retrieval-based systems, have achieved impres- sive results. While generally effective, these models fail to provide a satisfying answer when all retrieved …...