TDP – A Hybrid Diacritic Restoration with Transformer Decoder

TDP – A Hybrid Diacritic Restoration with Transformer Decoder DANG Trung Duc Anh ducanhhbtt@gmail.com Hanoi University of Science and Technology Hanoi, Vietnam NGUYEN Thi Thu Trang∗ trangntt@soict.hust.edu.vn Hanoi University of Science and Technology Hanoi, Vietnam Abstract Diacritic restoration plays an important role in Natural Language Processing (NLP) for many diacritical languages such as Vietnamese, Czech, Hungarian, etc. With the develop- ment of deep neural network, this task could reach a good accuracy, i.e. F1-score is up to 97.7% for the state-of-the-art models. How- ever, the output of these models can include meaningless syllables, the processing time is rather long especially with the sequence-to- sequence method and beam search. This task is a very first step in any text (pre-)processing, which can be a part of another application. Therefore, the processing time is extremely important. To balance both accuracy and time consuming, this paper proposes a novel hy- brid method which includes a transformer de- coder and a diacritic penalty layer. The trans- former decoder is good enough for this prob- lem since an input character only corresponds to exact one output character. The purpose of the penalty layer is to guide the model to produce only possible diacritic letters of the language. The experimental results on Viet- namese corpus show that the proposed model helps the predicting time reduce from about eight to ten times compared to the previous methods. Whereas, the accuracy of the pro- posed method is better than (i.e. 1%) or equal to the state-of-the-art sequence-to-sequence without or with beam search. ∗*Corresponding author 1 Introduction Diacritics are a vitally important component in many diacritical languages such as Vietnamese, Czech, Hungarian, etc. However, a large number of texts without diacritics serve many purposes. In diacritic languages, typing a diacritic word is far more troubled than typing a non-diacritic one. For instance, in Vietnamese, “đường” can be typed as "dduongwf" (Telex system). The middle or old- aged who do not know the rules for typing or those who want to save time typing diacritics would prefer to type sentences without diacritics, although they can be misleading and incomprehensible to other people. Moreover, many public-originated foreign systems only support non-diacritic characters, lead- ing to a huge amount of this data type to be pro- cessed. There are numerous ways for restoring a sentence to its former full diacritical marks with different meanings. For example, in Vietnamese, the most suitable restoration version for the non-diacritic one “Toi muon mo the tin dung” is "Tôi muốn mở thẻ tín dụng” (I want to open a credit card). However, each syllable in the sentence has multiple ways to become the form with full diacritic marks. "Muon" can be restored as "muốn" (want), "muộn" (late), "muôn" (many) while "the" can be restored as "thẻ" (card), "thè" (put out), "the" (a kind of traditional cloth), "thé" (high pitch), "thế" (and), "thề" (swear), "thê" (wife). This leads to a number of restoration com- binations corresponding to the input sentence that we need to disambiguate. Therefore, recovering …

Content truncated to 500 words