Adapting Language Models When Training on Privacy-Transformed Data

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4367–4373 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4367 Adapting Language Models When Training on Privacy-Transformed Data M. A. Tu˘gtekin Turan1, Dietrich Klakow2, Emmanuel Vincent1, and Denis Jouvet1 Universit´e de Lorraine, CNRS, Inria, Loria, F-54000, Nancy, France1 Spoken Language Systems Group, Saarland University, Germany2 {tugtekin.turan,emmanuel.vincent,denis.jouvet}@inria.fr dietrich.klakow@lsv.uni-saarland.de Abstract In recent years, voice-controlled personal assistants have revolutionized the interaction with smart devices and mobile applications. The collected data are then used by system providers to train language models (LMs). Each spoken message reveals personal information, hence removing private information from the input sentences is necessary. Our data sanitization process relies on recognizing and replacing named entities by other words from the same class. However, this may harm LM training because privacy-transformed data is unlikely to match the test distribution. This paper aims to fill the gap by focusing on the adaptation of LMs initially trained on privacy-transformed sentences using a small amount of original untransformed data. To do so, we combine class-based LMs, which provide an effective approach to overcome data sparsity in the context of n-gram LMs, and neural LMs, which handle longer contexts and can yield better predictions. Our experiments show that training an LM on privacy-transformed data result in a relative 11% word error rate (WER) increase compared to training on the original untransformed data, and adapting that model on a limited amount of original untransformed data leads to a relative 8% WER improvement over the model trained solely on privacy-transformed data. Keywords: class-based language modeling, privacy-preserving learning, language model adaptation, speech-to-text 1. Introduction Spoken dialogue systems aim to identify users’ intents expressed in natural language and then to satisfy the corresponding requests. In the first step of any system, the input utterance is recognized with an automatic speech recognizer (ASR). State-of-the-art data-driven ASR models are trained on large amounts of speech data collected from the users (Tang et al., 2004). With the growing public awareness exemplified by the European Union’s General Data Protection Regulation (GDPR), storing the user data raises serious privacy concerns (Nautsch et al., 2019). The user data might even contain critical information such as passwords, credit card numbers, or even health status. Therefore, spoken messages which contain sensitive information about the user characteristics should not be centralized in a single place. This paper uses a data sanitization approach that removes personal information relating to persons, locations, and organization names. These named entities are identified using an automatic named entity recognition method (Li et al., 2020). Then they are hidden using word-by-word replacement by random named entities from the same entity class. Adelani et al. (2020) prove that this word-by-word replacement strategy provides formal privacy guarantees in terms of differential privacy. The challenge is to ensure that, despite the sanitization, the performance of an ASR system trained on the sanitized data remains (almost) as good as that of an ASR system trained on original untransformed data. Unfortunately, …

Content truncated to 500 words