Evaluating the Morphosyntactic Well-formedness of Generated Texts

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7131–7150 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7131 Evaluating the Morphosyntactic Well-formedness of Generated Texts Adithya Pratapa,1* Antonios Anastasopoulos,2* Shruti Rijhwani,1 Aditi Chaudhary,1 David R. Mortensen,1 Graham Neubig,1 Yulia Tsvetkov3 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science, George Mason University 3Paul G. Allen School of Computer Science & Engineering, University of Washington {vpratapa,srijhwan,aschaudh,dmortens,gneubig}@cs.cmu.edu antonis@gmu.edu, yuliats@cs.washington.edu Abstract Text generation systems are ubiquitous in nat- ural language processing applications. How- ever, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L’AMBRE – a metric to evaluate the morphosyntactic well- formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract var- ious rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train ro- bust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translat- ing into morphologically-rich languages.1 1 Introduction A variety of natural language processing (NLP) ap- plications such as machine translation (MT), sum- marization, and dialogue require natural language generation (NLG). Each of these applications has a different objective and therefore task-specific eval- uation metrics are commonly used. For instance, reference-based measures such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and chrF (Popovi´c, 2015) are used to evaluate MT, ROUGE (Lin, 2004) is a metric widely used in summarization, and various task-based metrics are used in dialogue (Liang et al., 2020). Regardless of the downstream application, an important aspect of evaluating language generation systems is measuring the fluency of the generated text. In this paper, we propose a metric that can be used to evaluate the grammatical well-formedness of text produced by NLG systems.2 Our metric *Equal contribution 1Code and data are available at https://github.com/ adithya7/lambre. 2While grammatical well-formedness is often necessary for fluent text, it is not sufficient (Sakaguchi et al., 2016). PRON AUX ADJ NOUN VERB S.1 Ich werde lange Bücher lesen I-NOM.1SG will-1SG long-ACC.PL Book-ACC.PL read-PTCP S.2 *Ich werden langen Bücher lesen I-NOM.1SG will-1PL long-DAT.PL Book-ACC.PL read-PTCP root subj comp:aux comp:obj mod root ERROR comp:aux CORRECT ERROR number, person agreement number, case, gender agreement case assignment case assignment Figure 1: Identifying grammatical errors in text us- ing dependency parses and morpho-syntactic rules. Un- grammatical sentence S.2 fails to satisfy subject-verb agreement between PRON and AUX as well as case agreement between ADJ and NOUN. However, it sat- isfies case assignment rules with the subject in NOM case and the object in ACC case respectively. is referenceless and is based on the grammatical rules of the language, thereby enabling fine-grained identification and analysis of which grammatical phenomena the NLG system is struggling with. Although several referenceless metrics for evalu- ating NLG models exist, most use features of both the input and output, limiting their applicability to …

Content truncated to 500 words