Text Normalization Infrastructure that Scales to Hundreds of Language Varieties

Text Normalization Infrastructure that Scales to Hundreds of Language Varieties Mason Chua, Daan van Esch, Noah Coccaro, Eunjoon Cho, Sujeet Bhandari, Libin Jia Google LLC, 1600 Amphitheatre Parkway, Mountain View, CA 94043 {aftran, dvanesch, noahc}@google.com Abstract We describe the automated multi-language text normalization infrastructure that prepares textual data to train language models used in Google’s keyboards and speech recognition systems, across hundreds of language varieties. Training corpora are sourced from various types of data sets, and the text is then normalized using a sequence of hand-written grammars and learned models. These systems need to scale to hundreds or thousands of language varieties in order to meet product needs. Frequent data refreshes, privacy considerations and simultaneous updates across such a high number of languages make manual inspection of the normalized training data infeasible, while there is ample opportunity for data normalization issues. By tracking metrics about the data and how it was processed, we are able to catch internal data preparation issues and external data corruption issues that can be hard to notice using standard extrinsic evaluation methods. Showing the importance of paying attention to data normalization behavior in large-scale pipelines, these met- rics have highlighted issues in Google’s real-world speech recognition system that have caused significant, but latent, quality degradation. Keywords: language model, text normalization, internationalization, industrial systems, data mining, data processing, scale, less-resourced languages 1. Introduction As technology adoption increases, products need to sup- port ever more language varieties (ITU/UNESCO Broad- band Commission, 2017). For example, there is a clear need for spell-checking and keyboards in hundreds or even thousands of language varieties. These applications typi- cally require a language model, as do more advanced tech- nologies like speech recognition systems. It would not be feasible to maintain separate training pipelines for all these use cases multiplied by all these language varieties, so we use one unified pipeline that is flexible enough to train all the models. The training includes automatic, configurable preprocessing for each data set, allowing text from a variety of sources to be normalized into the domains appropriate for keyboard prediction and speech recognition (Schwarm and Ostendorf, 2002; Ju et al., 2008; Scannell, 2014). 2. Training Pipeline Each language model is produced by interpolation of component models, which are trained on individual data sources. The training set for the typical component model begins as publicly crawled or logs data. It is processed by language identification classifiers and privacy-enhancing systems (including for anonymization). The data is then filtered and edited by a sequence of normalizers (section 2.1) specific to the use case and language variety. The creation of the interpolated language model is con- trolled by a configuration file specifying how to find and normalize each data set, as well as global model options like interpolation method. The configuration system sep- arates language-specific data and its preprocessing from the training algorithms used. A single training binary can therefore train a model for any supported language, and new languages can be added simply by adding a configu- ration …

Content truncated to 500 words