Training Hybrid Language Models by Marginalizing over Segmentations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1477–1482 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1477 Training Hybrid Language Models by Marginalizing over Segmentations Edouard Grave Sainbayar Sukhbaatar Piotr Bojanowski Armand Joulin Facebook AI Research {egrave,sainbar,bojanowski,ajoulin}@fb.com Abstract In this paper, we study the problem of hy- brid language modeling, that is using models which can predict both characters and larger units such as character ngrams or words. Us- ing such models, multiple potential segmenta- tions usually exist for a given string, for ex- ample one using words and one using charac- ters only. Thus, the probability of a string is the sum of the probabilities of all the possi- ble segmentations. Here, we show how it is possible to marginalize over the segmentations efficiently, in order to compute the true prob- ability of a sequence. We apply our technique on three datasets, comprising seven languages, showing improvements over a strong character level language model. 1 Introduction Statistical language modeling is the problem of estimating a probability distribution over text data (Bahl et al., 1983). Most approaches formu- late this problem at the word level, by first seg- menting the text using a fixed vocabulary. A limi- tation of these methods is that they cannot generate new words, or process out of vocabulary words. A popular alternative is to directly model sequences at the character level. These models can poten- tially generate any sequence, and are thus some- times referred to as open vocabulary. However, they tend to underperform compared to word level models when trained on the same data. For these reasons, a few works have proposed hybrid models, that work both at the character and word level (or sometimes groups of characters). A first class of hybrid models switch between word and character level representations, depending on whether they predict that the upcoming word is in the vocabulary or not (Kawakami et al., 2017; Mielke and Eisner, 2019). For example, a first model can be trained on tokenized data, where out- of-vocabulary words are replaced by the <unk> token. A second model is then used to generate the character sequences corresponding to out-of- vocabulary words. Another approach, which does not require tokenization, is to process groups of characters, which are obtained based on linguis- tic knowledge or low level statistics. These in- clude merging characters using mutual informa- tion (Mikolov et al., 2012) or the byte pair en- coding algorithm (Sennrich et al., 2016). This ap- proach first produces a segmentation for the text, and then learns a language model on it. However, some sequences have multiple possible segmenta- tions, and a model considering a single one might underestimate the true probability of the sequence. Thus, it is important to marginalize over the set of segmentations to obtain the true probability of a sequence (van Merri¨enboer et al., 2017; Buckman and Neubig, 2018). In this paper, we propose an alternative ap- proach to address this limitation, and …

Content truncated to 500 words