Automated Evaluation of Out-of-Context Errors

Automated Evaluation of Out-of-Context Errors Patrick Huber, Jan Niehues, Alex Waibel Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology (KIT) uhejt@student.kit.edu, jan.niehues@kit.edu, alex.waibel@kit.edu Abstract We present a new approach to evaluate computational models for the task of text understanding by the means of out-of-context error detection. Through the novel design of our automated modification process, existing large-scale data sources can be adopted for a vast number of text understanding tasks. The data is thereby altered on a semantic level, allowing models to be tested against a challenging set of modified text passages that require to comprise a broader narrative discourse. Our newly introduced task targets actual real-world problems of transcription and translation systems by inserting authentic out-of-context errors. The automated modification process is applied to the 2016 TEDTalk corpus. Entirely automating the process allows the adoption of complete datasets at low cost, facilitating supervised learning procedures and deeper networks to be trained and tested. To evaluate the quality of the modification algorithm a language model and a supervised binary classification model are trained and tested on the altered dataset. A human baseline evaluation is examined to compare the results with human performance. The outcome of the evaluation task indicates the difficulty to detect semantic errors for machine-learning algorithms and humans, showing that the errors cannot be identified when limited to a single sentence. Keywords: Out-of-Context Error Recognition, Automatic Evaluation Dataset, Text Understanding, TEDTalk 1. Introduction Machine learning strives to achieve universal application in solving arbitrary tasks rather than special functionalities and well-defined functions. This trend is reflected by the in- creasing importance of text understanding within the field of natural language processing (NLP). Extracting the context of a narrative passage can thereby benefit a wide variety of applications, such as automatic speech recognition (ASR) and neural machine translation (NMT) tasks. Most state-of-the-art ASR and NMT sys- tems are currently only processing one sentence at a time, solely basing the system’s decisions on the local context (Bahdanau et al., 2014; Cho et al., 2014). This approach is reasonable, as it addresses the most general case. Never- theless, considering a larger context can highly benefit the overall system’s performance for text passages with contex- tual features. Imagine an ASR system that is supposed to transcribe the sentence There are great opportunities in the far-east within a presentation on the global economy. Through an inarticulate pronunciation and a noisy signal at the confer- ence, the ASR system transcribes the sentence as There are great opportunities in the forest, as the sentences are pho- netically very similar. In this example, the language model of the ASR system will not be able to identify the out-of- context error by only considering the current sentence. Therefore, developing a context component to enhance the text understanding is an important task to improve state-of- the-art systems. While the number of published datasets regarding question- answering (QA) tasks steadily increases, for example based on children books (Hill et al., 2015) or simple reasoning (Weston et al., 2015), there …

Content truncated to 500 words