WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3661–3672 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3661 WER–BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm Akshay Krishna Sheshadri* Anvesh Rao Vijjini* Samsung R&D Institute India - Bangalore {a.sheshadri,a.vijjini,sukhdeep.sk}@samsung.com Sukhdeep Kharbanda Abstract Automatic Speech Recognition (ASR) sys- tems are evaluated using Word Error Rate (WER), which is calculated by comparing the number of errors between the ground truth and the transcription of the ASR system. This cal- culation, however, requires manual transcrip- tion of the speech signal to obtain the ground truth. Since transcribing audio signals is a costly process, Automatic WER Evaluation (e- WER) methods have been developed to auto- matically predict the WER of a speech system by only relying on the transcription and the speech signal features. While WER is a con- tinuous variable, previous works have shown that positing e-WER as a classification prob- lem is more effective than regression. How- ever, while converting to a classification set- ting, these approaches suffer from heavy class imbalance. In this paper, we propose a new balanced paradigm for e-WER in a classifica- tion setting. Within this paradigm, we also propose WER-BERT, a BERT based architec- ture with speech features for e-WER. Further- more, we introduce a distance loss function to tackle the ordinal nature of e-WER classifica- tion. The proposed approach and paradigm are evaluated on the Librispeech dataset and a commercial (black box) ASR system, Google Cloud’s Speech-to-Text API. The results and experiments demonstrate that WER-BERT es- tablishes a new state-of-the-art in automatic WER estimation. 1 Introduction ASR systems are ubiquitous now. They are avail- able across applications such as Voice Assistants, Assisted Living or Hands free device usage. How- ever, with the widespread usage of ASR systems, there comes a heavy need for ASR Evaluation as well - to select, compare or improve alternate ASR *The authors contributed equally to the work. systems. WER is widely considered as the standard metric for ASR evaluation. A higher WER means a higher percentage of errors between the ground truth and the transcription from the system. WER is calculated by aligning the two text segments us- ing string alignment in a dynamic programming setting. The formula is as follows: WER = ERR N (1) where ERR is the sum of errors (Insertions, Deletions or Substitutions) between the transcrip- tion and the ground truth. N is number of words in ground truth. As evident from this equation, the presence of ground truth is necessary for the cal- culation of errors, and hence, for WER. However, manual transcription of speech at word level is a expensive and dilatory process. Hence, the need for an automatic ASR evaluation is important but few attempts have addressed this. Furthermore, for effectively training, evaluating and judging the performance of an ASR system, WER calculation needs to be done on adequate hours of data. As this test set increases, a more accurate …

Content truncated to 500 words