PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4611–4619 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4611 PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics Jordan Meadows1,2, Zili Zhou1, Andr´e Freitas1,2 1Department of Computer Science, University of Manchester 2Idiap Research Institute, Switzerland jordan.meadows@postgrad.manchester.ac.uk {zili.zhou, andre.freitas}@manchester.ac.uk Abstract In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines that demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives. Keywords: mathematical text, physics, natural language understanding, discourse coherence 1. Introduction Physics literature is a form of mathematical language which is unique beyond simply domain vocabulary. How physi- cists use mathematics to reason and explain, separates their field fundamentally from other disciplines, including math- ematics. Many of its sub-disciplines are situated between pure mathematics and engineering, while others conjoin computer science and biology, with physical methods act- ing as a well-travelled bridge between the formal and nat- ural sciences. It has not been proven, for example, that smooth solutions (Pizzocchero, 2021; Gala et al., 2021; Miller, 2021) always exist for the Navier-Stokes equations (a millenium problem) despite their widespread use in sim- ulating and engineering fluid dynamics, while biophysics demonstrates that fundamental problems in ecology and evolution can be characterized by computational complex- ity classes (Ibsen-Jensen et al., 2015). Physics discourse serves as a universal mechanism for generating empirically falsifiable quantitative theory in the natural sciences and en- gineering (Smith and Fleck, 2017; Coffey and Kalmykov, 2012), separate to both pure mathematics and any down- stream field. Its core traits are reflected in unique literary devices and its mathematical explanations will differ to those in the formal sciences as a result. A concrete example is the physics derivation; a core explanatory or argumentative de- vice less rigorous and more informal than mathematical proofs (Meadows and Freitas, 2021; Davis, 2019; Kaliszyk et al., 2015), which generally results in predictive equations relating physical quantities, rather than generating a truth value for a given conjecture (e.g., twin primes). Such equa- tions are central components of physics descriptions, with natural language forming around them and their elements, and their relation to other equations through derivations. Mathematics as a whole, particularly logic, is less con- cerned with this predictive modelling of real world sys- tems, let alone when such systems are quantum or rela- tivistic, or both. Suggesting that mathematicians work at a level of abstraction higher than that …

Content truncated to 500 words