Text Normalization

This paper addresses the issue of text nor- malization, an important yet often over- looked problem in natural language proc- essing. By text normalization, we mean converting ‘informally inputted’ text …...

We present a novel method for discovering and modeling the relationship between in- formal Chinese expressions (including collo- quialisms and instant-messaging slang) and their formal equivalents. Specifically, we pro- posed …...

AbstractPersian with its about 100,000,000 speakers in the world belongs to the group of languages with less developed linguistically annotated resources and tools. The few existing resources and tools are …...

There are many abbreviation and non-standard tokens in SMS and Twitter messages. Nor- malizing these non-standard tokens will ease natural language processing modules for these domains. Recently, character-level machine translation …...

Social media texts are written in an infor- mal style, which hinders other natural lan- guage processing (NLP) applications such as machine translation. Text normalization is thus important for processing …...

We present a unified unsupervised statistical model for text normalization. The relation- ship between standard and non-standard to- kens is characterized by a log-linear model, permitting arbitrary features. The weights …...

Text normalization is an important first step towards enabling many Natural Lan- guage Processing (NLP) tasks over infor- mal text. While many of these tasks, such as parsing, perform the …...

The informal nature of social media text renders it very difficult to be automati- cally processed by natural language pro- cessing tools. Text normalization, which corresponds to restoring the non-standard …...

AbstractWe present a dependency conversion of five German test sets from five different genres. The dependency representation is made as similar as possible to the dependency representation of TiGer, one …...

. Analogy-based Text Normalization : the case of unknowns words. In this paper, we describe and evaluate a system for improving the quality of noisy texts containing non-word errors. It …...

AbstractIn this paper we present a Dutch and English dataset that can serve as a gold standard for evaluating text normalization approaches. With the combination of text messages, message board …...

We propose a transition-based model for joint word segmentation, POS tagging and text normalization. Different from pre- vious methods, the model can be trained on standard text corpora, overcoming the …...

Recent years have seen increased interest in text normalization in social media, as the in- formal writing styles found in Twitter and other social media data often cause problems for …...

AbstractWe introduce the Denoised Web Treebank: a treebank including a normalization layer and a corresponding evaluation metric for dependency parsing of noisy text, such as Tweets. This benchmark enables the …...

AbstractWe present a text-to-speech (TTS) system designed for the dialect of Bengali spoken in Bangladesh. This work is part of an ongoing effort to address the needs of under-resourced languages. …...

AbstractAttention-based sequence-to-sequence neural network models learn to jointly align and translate. The quadratic-time attention mechanism is powerful as it is capable of handling arbitrary long-distance reordering, but computationally expensive. In …...

We describe the automated multi-language text normalization infrastructure that prepares textual data to train language models used in Google’s keyboards and speech recognition systems, across hundreds of language varieties. Training …...

AbstractOne of the main characteristics of social media data is the use of non-standard language. Since NLP tools have been trained on traditional text material their performance drops when applied …...

AbstractArabic dialects are the non-standard varieties of Arabic commonly spoken – and increasingly written on social media – across the Arab world. Arabic dialects do not have standard orthographies, a …...

AbstractWe propose neural models that can normalize text by considering the similarities of word strings and sounds. We experimentally compared a model that considers the similarities of both word strings …...

AbstractWe propose Seq2Edits, an open-vocabulary approach to sequence editing for natural language processing (NLP) tasks with a high degree of overlap between input and output texts. In this approach, each …...

AbstractCross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. However, they currently require large pretraining corpora or access to typologically …...

AbstractThis paper details the Johns Hopkins speech translation (ST) system used in the IWLST2022 dialect speech translation task. Our system uses a cascade of automatic speech recognition (ASR) and machine …...