Efficient Convolutional Neural Networks for Diacritic Restoration

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1442–1448, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1442 Efficient Convolutional Neural Networks for Diacritic Restoration Sawsan Alqahtani∗2,3 and Ajay Mishra1 and Mona Diab1,2 1AWS, Amazon AI 2The George Washington University 3Princess Nourah Bint Abdul Rahman University sawsanq@gwu.edu, misaja@amazon.com, diabmona@amazon.com Abstract Diacritic restoration has gained importance with the growing need for machines to under- stand written texts. The task is typically mod- eled as a sequence labeling problem and cur- rently Bidirectional Long Short Term Mem- ory (BiLSTM) models provide state-of-the-art results. Recently, Bai et al. (2018) show the advantages of Temporal Convolutional Neu- ral Networks (TCN) over Recurrent Neural Networks (RNN) for sequence modeling in terms of performance and computational re- sources. As diacritic restoration benefits from both previous as well as subsequent timesteps, we further apply and evaluate a variant of TCN, Acausal TCN (A-TCN), which incor- porates context from both directions (previ- ous and future) rather than strictly incorporat- ing previous context as in the case of TCN. A-TCN yields significant improvement over TCN for diacritization in three different lan- guages: Arabic, Yoruba, and Vietnamese. Fur- thermore, A-TCN and BiLSTM have compa- rable performance, making A-TCN an effi- cient alternative over BiLSTM since convo- lutions can be trained in parallel. A-TCN is significantly faster than BiLSTM at infer- ence time (270%∼334% improvement in the amount of text diacritized per minute). 1 Introduction A diacritic is a mark that is added above, below, or within a letter, constructing a new letter or charac- terizing it with a different sound (Wells, 2000). In languages such as Arabic and Hebrew, some vowels are not part of the alphabet and diacritics are used for vowel specification. Moreover, many languages that include diacritics such as Yoruba and Vietnamese sometimes omit them in writing ∗*The work was done while the author was an intern at AWS, Amazon AI. for various reasons such as difficulty in typing di- acritics on keyboards or digitizing electronic text (Scannell, 2011). Arabic, on the other hand, considers diacritics as an optional choice in writing even in formal settings,1 and familiarity with the language is re- lied upon to derive the meaning based on context. Although people can extrapolate missing diacrit- ics with near perfect accuracy in such languages, missing diacritics pose a challenge for computa- tional models due to increased ambiguity. Diacritization is the process of automatically restoring missing diacritics for each character in written text. It is important in many NLP ap- plications such as automatic speech recognition (Vergyri and Kirchhoff, 2004) and speech synthe- sis (Ungurean et al., 2008). Most state-of-the- art diacritization models use Bidirectional Long Short Term Memory (BiLSTM) (Hochreiter and Schmidhuber, 1997) as a sequential classification problem, or as a sequence-to-sequence transduc- tion problem to convert the original text into dia- critized form (Orife, 2018; Zalmout and Habash, 2017; Belinkov and Glass, 2015). Generally speaking, LSTM …

Content truncated to 500 words