Tree-Based Representation and Generation of Natural and Mathematical Language

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3714–3730 July 9-14, 2023 ©2023 Association for Computational Linguistics Tree-Based Representation and Generation of Natural and Mathematical Language Alexander Scarlatos and Andrew Lan University of Massachusetts Amherst {ajscarlatos,andrewlan}@cs.umass.edu Abstract Mathematical language in scientific communi- cations and educational scenarios is important yet relatively understudied compared to natural languages. Recent works on mathematical lan- guage focus either on representing stand-alone mathematical expressions, especially in their natural tree format, or mathematical reasoning in pre-trained natural language models. Ex- isting works on jointly modeling and generat- ing natural and mathematical languages simply treat mathematical expressions as text, without accounting for the rigid structural properties of mathematical expressions. In this paper, we propose a series of modifications to existing language models to jointly represent and gen- erate text and math: representing mathemati- cal expressions as sequences of node tokens in their operator tree format, using math sym- bol and tree position embeddings to preserve the semantic and structural properties of math- ematical expressions, and using a constrained decoding method to generate mathematically valid expressions. We ground our modifica- tions in GPT-2, resulting in a model MathGPT, and demonstrate that it outperforms baselines on mathematical expression generation tasks. 1 Introduction A part of human communication is performed in rigorous mathematical language rather than more flexible natural language, which often occurs in sce- narios such as scientific communication and educa- tion. While pre-trained large language models such as BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020) have enjoyed many successes in repre- senting and generating natural language, there is a need for models that are effective in representing and generating principled mathematical language as well. While existing work focuses on various aspects of mathematical language representation or generation, combining mathematical language with the aforementioned models for natural language re- mains a challenging problem. Mathematical and natural language are funda- mentally different in many ways. While natural language consists of large sets of words and phrases that often have their meaning grounded in context, mathematical language consists of different sym- bols: a small set of mathematical operators with precise meaning, variables, and numbers that exist in a continuous space. Furthermore, mathematical language follows rules that are much more strict and rigorous than natural language. For example, the multiplication operation acts on exactly two operands, while an integral operates on a single operand but with upper and lower limit arguments. Operands are either variables, numbers, or the re- sult of applying other operations. Given its hierar- chical nature, mathematical language is naturally represented with operator trees (OPTs), which are directed tree graphs where non-leaf nodes are op- erators and leaf nodes are variables or numbers (Zanibbi et al., 2016a; Mansouri et al., 2019). OPTs are effective at capturing both the semantic and structural properties of mathematical expressions. Existing work primarily focuses on two separate approaches to modeling mathematical language: representation and mathematical reasoning. A line of work focuses on …

Content truncated to 500 words