Controlling Utterance Length in NMT-based Word Segmentation with Attention

Controlling Utterance Length in NMT-based Word Segmentation with Attention Pierre Godard1,2, Laurent Besacier2, Franc¸ois Yvon1 (1) LIMSI, CNRS, Universit´e Paris-Saclay, F 91 405 Orsay (2) LIG, CNRS et Universit´e Grenoble Alpes, F 38 000 Grenoble pierre.godard@limsi.fr, laurent.besacier@uga.fr, francois.yvon@limsi.fr Abstract One of the basic tasks of computational language doc- umentation (CLD) is to identify word boundaries in an un- segmented phonemic stream. While several unsupervised monolingual word segmentation algorithms exist in the lit- erature, they are challenged in real-world CLD settings by the small amount of available data. A possible remedy is to take advantage of glosses or translation in a foreign, well- resourced, language, which often exist for such data. In this paper, we explore and compare ways to exploit neural ma- chine translation models to perform unsupervised boundary detection with bilingual information, notably introducing a new loss function for jointly learning alignment and segmen- tation. We experiment with an actual under-resourced lan- guage, Mboshi, and show that these techniques can effec- tively control the output segmentation length. 1. Introduction All over the world, languages are disappearing at an unprece- dented rate, fostering the need for specific tools aimed to aid field linguists to collect, transcribe, analyze, and annotate en- dangered language data (e.g. [1, 2]). A remarkable effort in this direction has improved the data collection procedures and tools [3, 4], enabling to collect corpora for an increasing number of endangered languages (e.g. [5]). One of the basic tasks of computational language doc- umentation (CLD) is to identify word or morpheme bound- aries in an unsegmented phonemic or orthographic stream. Several unsupervised monolingual word segmentation al- gorithms exist in the literature, based, for instance, on information-theoretic [6, 7] or nonparametric Bayesian tech- niques [8, 9]. These techniques are, however, challenged in real-world settings by the small amount of available data. A possible remedy is to take advantage of glosses or translations in a foreign, well-resourced language (WL), which often exist for such data, hoping that the bilingual con- text will provide additional cues to guide the segmentation algorithm. Such techniques have already been explored, for instance, in [10, 11] in the context of improving statistical alignment and translation models; and in [12, 13, 14] using Attentional Neural Machine Translation (NMT) models. In these latter studies, word segmentation is obtained by post- processing attention matrices, taking attention information as a noisy proxy to word alignment [15].1 In this paper, we explore ways to exploit neural machine translation models to perform unsupervised boundary detec- tion with bilingual information. Our main contribution is a new loss function for jointly learning alignment and segmen- tation in neural translation models, allowing us to better con- trol the length of utterances. Our experiments with an actual under-resourced language (UL), Mboshi [18], show that this technique outperforms our bilingual segmentation baseline. 2. Recurrent architectures in NMT In this section, we briefly review the main concepts of re- current architectures for machine translation introduced in [19, 20, 21].2 In our setting, the source and target sentences are …

Content truncated to 500 words