HPSG/MRS-Based Natural Language Generation Using Transformer

HPSG/MRS-Based Natural Language Generation Using Transformer Gyu-min Lee Department of Linguistics Korea University gyuminlee@korea.ac.kr Sanghoun Song Department of Linguistics Korea University sanghoun@korea.ac.kr Abstract With the massive success of stochastic lan- guage models, some claim that the symbolic approach is not necessary for natural language processing. However, recent studies made use of linguistically motivated meaning representa- tion of Abstract Meaning Representation and Minimal Recursion Semantics in a natural lan- guage generation task and achieved impressive results while hinting that neural networks do make extensive use of linguistic information. This study partially replicates previous studies with Transformer. The model performed worse than the previous researches. We present some evidence that the model failed to make the cor- rect lexical choices due to Attention weight assignment. Overall, it appears that models built solely on Attention mechanism are sub- optimal for processing language representation with rich grammatical information, while hint- ing the possible use of engineered grammar for improving the controllability of the neural models. 1 Introduction While symbolic natural language processing (NLP) models have great precision, they have limited cov- erage. Meanwhile, stochastic NLP models, notably neural NLP, boast broad coverage but lack control- lability. Then, is it possible to join the two frame- works and improve a model’s capacity, precision, and controllability at the same time? The current study explores a way to integrate the two approaches by replicating previous researches with Transformer (Vaswani et al., 2017) and finds out that Attention- dependant approaches may not be optimal for pro- cessing grammatical representations. Modern NLP is largely stochastic. From statistical language processors to the neural language models, NLP rapidly moved from its symbolic, algorithm based predecessors. When it comes to the empirical evaluation, the stochastic NLP models are performing much better while being cheaper to design concep- tually. In fact, stochastic NLP models perform so well that some question the necessity of symbolic knowledge of linguistics itself in NLP. However, neural language models inevitably suffer from the “black box” problem. In other words, we do not really know how they work. Therefore, de- bugging the neural language models and make them perfect is a difficult process. Some researchers solve this problem by increasing model sizes. However, its environmental impact cannot be neglected, needless to say whether it is ever possible for such gigantic language models can indeed “learn” the language (Bender et al., 2021). On the other side of the coin, it is possible to engi- neer a computational grammar out of certain linguis- tic formalism. Called grammar engineering (Bender and Emerson, 2021; Bender et al., 2008), this pro- cess bore several computational grammars including English Resource Grammar (ERG; Flickinger, 2000), which is an English grammar based on the Head- Driven Phrase Structure Grammar (HPSG; Pollard and Sag, 1994) framework. While such grammars show high precision, they show low recall (Bender and Emerson, 2021), indicating that their coverage is limited. Hajdik et al. (2019) integrates neural natural lan- guage generation (NLG) with ERG. In detail, they defined NLG as a task of …

Content truncated to 500 words