The Effect of Dependency Representation Scheme on Syntactic Language Modelling

The Effect of Dependency Representation Scheme on Syntactic Language Modelling Sunghwan Mac Kim Department of Computing Macquarie University Sydney, NSW 2109, Australia sunghwan.kim@mq.edu.au John K Pate Department of Computing Macquarie University Sydney, NSW 2109, Australia john.pate@mq.edu.au Mark Johnson Department of Computing Macquarie University Sydney, NSW 2109, Australia mark.johnson@mq.edu.au Abstract There has been considerable work on syn- tactic language models and they have ad- vanced greatly over the last decade. Most of them have used a probabilistic context- free grammar (PCFG) or a dependency grammar (DG). In particular, DG has at- tracted more and more interest in the past years since dependency parsing has achieved great success. While much work has evaluated the effects of different de- pendency representations in the context of parsing, there has been relatively little in- vestigation into them on a syntactic lan- guage model. In this work, we conduct the first assessment of three dependency representations on a transition-based de- pendency parsing language model. We show that the choice of dependency rep- resentation has an impact on overall per- formance from the perspective of language modelling. 1 Introduction Syntactic language models have been successfully applied to a wide range of domains such as speech recognition, machine translation, and disfluency detection. Although n-gram based language mod- els are the most widely used due to their simplic- ity and efficacy, they suffer from a major draw- back: they cannot characterise the long-range rela- tions between words. A syntactic language model, which exploits syntactic dependencies, can incor- porate richer syntactic knowledge and information through syntactic parsing. In particular, syntactic structure leads to better performance of language model compared to traditional n-gram language models (Chelba and Jelinek, 2000). Most of the syntactic language models have used a probabilis- tic context-free grammar (PCFG) (Roark, 2001; Charniak, 2001) or a dependency grammar (DG) (Wang and Harper, 2002; Gubbins and Vlachos, 2013) in order to capture the surface syntactic structures of sentences. Researchers have shown an increased interest in dependency parsing and it has increasingly been recognised as an alternative to constituency pars- ing in the past years. Accordingly, various repre- sentations and associated parsers have been pro- posed with respect to DG. DG describes the syn- tactic structure of a sentence in terms of head- dependent relations between words. Unlike con- stituency models of syntax, DG directly models relationships between pairs of words, which leads to a simple framework that is easy to lexicalise and parse with. Moreover, a DG-based, particularly transition-based, parser is fast and even achieves state-of-the-art performance compared to the other grammar-based parsers. It is therefore suitable for identifying syntactic structures in incremental pro- cessing, which is a useful feature for online pro- cessing tasks such as speech recognition or ma- chine translation. The aim of this study is to explore different de- pendency representations and to investigate their effects on the language modelling task. There are publicly available converters to generate depen- dencies. Ivanova et al. (2013) investigated the ef- fect of each dependency scheme in terms of …

Content truncated to 500 words