Improving Automatic Evaluation of Acceptability Based on Language Models with a Coarse Sentence Representation

Improving Automatic Evaluation of Acceptability Based on Language Models with a Coarse Sentence Representation Vijay Daultani Tokyo Institute of Technology Meguro City, Tokyo vijay.daultani@nlp.c.titech.ac.jp Naoaki Okazaki Tokyo Institute of Technology Meguro City, Tokyo okazaki@c.titech.ac.jp Abstract Motivated by recent findings on the proba- bilistic modeling of the acceptability judg- ments, several metrics have been proposed for its automatic evaluation. Frequently used met- rics such as syntactic log odds ratio (SLOR) and its variants are based on utilizing proba- bility from language model (LM) as a proxy for automatic acceptability evaluation of the generated text from an LM. Since one can- not use probability directly as a measure of acceptability, these metrics take steps to re- move the confounding effects of noise from sentence length and lexical frequency to en- able the usage of probability for acceptability evaluation. In this work, we argue that even though the effects are reduced, they still ex- ist. We propose a data transformation strategy, Replace Named Entity (RNE), to get a coarse representation of a sentence to mitigate the re- maining problems from lexical frequency. In RNE, we identify all proper nouns (i.e., NEs) in a sentence and classify them into one of eighteen types. Later RNE replaces all occur- rences of NEs in a sentence with their identi- fied type. We later trained three LMs (2, 3, 4- grams) and assessed their performance of five acceptability measures on four test datasets. We found that LMs trained on datasets pre- processed by RNE yield a significantly higher correlation (upto 52% on some datasets) with human acceptability judgment. 1 Introduction Language Models (LMs) are often used to gener- ate natural language text for NLP tasks — Machine Translation, Summarization, Question Answering, and many others. Moreover, intrinsic evaluation of the LMs often includes at least two characteristics (Mutton et al., 2007). First, how well the generated text represents the source data, whether it be the text in another language for machine translation, text to represent a summary of a document, or text to rep- resent answers for a question, etc. Second, how well it conforms to regular human language use, a prop- erty we will refer to as acceptability of the sentence. Acceptability evaluation of a sentence is an essen- tial task for automatically evaluating the quality of the text (to help filter unacceptable sentences) gen- erated by the LMs. Before moving forward, it is also essential to un- derstand the difference between the usage of related words, i.e., fluency, readability, and grammaticality. Both fluency and readability are alternate words for acceptability, but the exact definition of these terms varies across the literature (Lau et al., 2017; Mut- ton et al., 2007; Kann et al., 2018; Storch, 2009; Pitler and Nenkova, 2008; Vadlapudi and Katra- gadda, 2010). However, we would like to differ- entiate acceptability from grammaticality. When a human evaluates the acceptability of a text, gram- maticality is one of the possible factors, among oth- ers like semantic plausibility, processing difficul- ties, etc., that can also influence …

Content truncated to 500 words