Tokenization

This paper introduces to the finite-state calculus a family of directed replace op- erators. In contrast to the simple re- place expression, UPPER -> LOWER, defined in Karttunen (1995), the …...

This paper presents a technique to deal with multiword nominal terminology in a compu- tational Lexical Functional Grammar. This method treats multiword terms as single to- kens by modifying the …...

This paper proposes a framework of language inde- pendent morphological analysis and mainly concen- trate on tokenization, the first process of morpholog- ical analysis. Although tokenization is usually not regarded …...

Entity relation detection is a form of in- formation extraction that finds predefined relations between pairs of entities in text. This paper describes a relation detection approach that combines clues …...

We describe a word alignment platform which ensures text pre-processing (to- kenization, POS-tagging, lemmatization, chunking, sentence alignment) as re- quired by an accurate word alignment. The platform combines two different …...

Training a statistical machine translation starts with tokenizing a parallel corpus. Some languages such as Chinese do not in- corporate spacing in their writing system, which creates a challenge for …...

As tokenization is usually ambiguous for many natural languages such as Chinese and Korean, tokenization errors might po- tentially introduce translation mistakes for translation systems that rely on 1-best to- …...

Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that …...

Most NLP systems use tokenization as part of preprocessing. Generally, tokenizers are based on simple heuristics and do not recog- nize multi-word units (MWUs) like hot dog or black hole …...

AbstractPersian with its about 100,000,000 speakers in the world belongs to the group of languages with less developed linguistically annotated resources and tools. The few existing resources and tools are …...

This paper presents a range of preprocess- ing solutions for Hebrew-English statisti- cal machine translation. Our best system, using a morphological analyzer, increases 3.5 BLEU points over a no-tokenization baseline …...

AbstractThis paper describes the creation process of an Indonesian-English parallel corpus (IDENTIC). The corpus contains 45,000 sentences collected from different sources in different genres. Several manual text preprocessing tasks, such …...

AbstractThe aim of this paper is to present a system for semantic text annotation called Inforex. Inforex is a web-based system designed for managing and annotating text corpora on the …...

AbstractThis article describes the collecting, processing and validation of a large balanced corpus for Romanian. The annotation types and structure of the corpus are briefly reviewed. It was constructed at …...

The many differences between Dialectal Ara- bic and Modern Standard Arabic (MSA) pose a challenge to the majority of Arabic natural language processing tools, which are designed for MSA. In …...

AbstractWe present our ongoing effort to create a massively parallel Bible corpus. While an ever-increasing number of Bible translations is available in electronic form on the internet, there is no …...

AbstractIn the last few years the amount of manuscripts digitized and made available on the Web has been constantly increasing. However, there is still a considarable lack of results concerning …...

This paper describes SWATAC, a system built for SemEval-2015’s Task 10 Subtask B, namely the Message Polarity Classification Task. Given a tweet, the system classifies the sentiment as either positive, …...

This paper contains the description of our par- ticipation at task 4 (sub-task A, Message Po- larity Classification) of SemEval-2016. Our proposed system consists mainly of three steps. Firstly, the …...

AbstractThe article describes the current status of a large national project, CoRoLa, aiming at building a reference corpus for the contemporary Romanian language. Unlike many other national corpora, CoRoLa contains …...

AbstractAutomatic natural language processing of large texts often presents recurring challenges in multiple languages: even for most advanced tasks, the texts are first processed by basic processing steps – from …...

AbstractIn this paper we present two deep-learning systems that competed at SemEval-2017 Task 4 “Sentiment Analysis in Twitter”. We participated in all subtasks for English tweets, involving message-level and topic-based …...

AbstractThe field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to “the” BLEU score, BLEU is …...

This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2.1 corpus (UD2) (Nivre et al., 2017). A software tool, which relies on Elephant …...

With the growth of Internet usage, a massive amount of textual data is generated on social media and the Web. As the text on the Web are generated by different …...

Tokenization, or morphological analysis, is a fundamental and important technology for processing a Japanese text, especially for in- dustrial applications. However, we often face many obstacles, such as the inconsistency …...

AbstractFor unsegmented languages such as Japanese and Chinese, tokenization of a sentence has a significant impact on the performance of text classification. Sentences are usually segmented with words or subwords …...

AbstractTypically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define …...

AbstractNegation is a universal linguistic phenomenon with a great qualitative impact on natural language processing applications. The availability of corpora annotated with negation is essential to training negation processing systems. …...

AbstractCross-lingual word embedding (CWE) algorithms represent words in multiple languages in a unified vector space. Multi-Word Expressions (MWE) are common in every language. When training word embeddings, each component word …...

AbstractTokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word tokenization to general …...

AbstractThis system paper describes an end-to-end NMT pipeline for the Japanese ↔ English news translation task as submitted to WMT 2021, where we explore the efficacy of techniques such as …...

AbstractIn this paper we present the final result of a project focused on Tunisian Arabic encoded in Arabizi, the Latin-based writing system for digital conversations. The project led to the …...

The grammatical analysis of texts in any written language typically involves a num- ber of basic processing tasks, such as tok- enization, morphological tagging, and de- pendency parsing. State-of-the-art systems …...

AbstractClinical Trials Reports (CTRs) contain highly valuable health information from which Natural Language Inference (NLI) techniques determine if a given hypothesis can be inferred from a given premise. CTRs are …...

AbstractWith the ever-growing amount of textual data, extractive summarization has become increasingly crucial for efficiently processing information. The TextRank algorithm, a popular unsupervised method, offers excellent potential for this task. …...