Higher-Order Syntactic Attention Network for Longer Sentence Compression

Proceedings of NAACL-HLT 2018, pages 1716–1726 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Higher-order Syntactic Attention Network for Long Sentence Compression Hidetaka Kamigaito♡, Katsuhiko Hayashi♢, Tsutomu Hirao♠ and Masaaki Nagata♠ ♡ Institute of Innovative Research, Tokyo Institute of Technology ♢ Institute of Scientific and Industrial Research, Osaka University ♠ NTT Communication Science Laboratories, NTT Corporation kamigaito@lr.pi.titech.ac.jp, katsuhiko-h@sanken.osaka-u.ac.jp, {hirao.tsutomu,nagata.masaaki}@lab.ntt.co.jp Abstract Sentence compression methods based on LSTM can generate fluent compressed sentences. However, the performance of these methods is significantly degraded when compressing long sentences since it does not explicitly handle syntactic fea- tures. To solve this problem, we propose a higher-order syntactic attention network (HiSAN) that can handle higher-order de- pendency features as an attention distri- bution on LSTM hidden states. Further- more, to avoid the influence of incorrect parse results, we train HiSAN by max- imizing the probability of a correct out- put together with the attention distribu- tion. Experiments on the Google sentence compression dataset show that our method achieved the best performance in terms of F1 as well as ROUGE-1,2 and L scores, 83.2, 82.9, 75.8 and 82.7, respectively. In subjective evaluations, HiSAN outper- formed baseline methods in both readabil- ity and informativeness. 1 Introduction Sentence compression is the task of compress- ing long sentences into short and concise ones by deleting words. To generate compressed sentences that are grammatical, many researchers (Jing, 2000; Knight and Marcu, 2000; Berg-Kirkpatrick et al., 2011; Filippova and Altun, 2013) have adopted tree trimming methods. Even though Fil- ippova and Altun (2013) reported the best results on this task, automatic parse errors greatly degrade the performances of these tree trimming methods. 1We used an LSTM-based sentence compression method (Filippova et al., 2015) in the evaluation setting as described in Section 4.1. 11-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 51-55 Sentence Length 60 65 70 75 80 85 90 F1 Figure 1: F1 scores of LSTM-based sentence com- pression method for each sentence length.1 11-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 51-55 Sentence Length 87654 9 10 11 12 Average Depth Figure 2: Average tree depths for each sentence length.2 Recently, Filippova et al. (2015) proposed an LSTM sequence-to-sequence (Seq2Seq) based sentence compression method that can generate fluent sentences without utilizing any syntactic features. Therefore, Seq2Seq based sentence com- pression is a promising alternative to tree trim- ming. However, as reported for a machine transla- tion task (Cho et al., 2014; Pouget-Abadie et al., 2014; Koehn and Knowles, 2017), the longer the input sentences are, the worse the Seq2Seq per- formances become. We also observed this prob- lem in the sentence compression task. As shown in Figure 1, the performance of Seq2Seq is de- graded when compressing long sentences. In par- ticular, the performance significantly falls if sen- tence length exceeds 26 words. This is an impor- tant problem, because sentences longer than the average sentence length (=28 words) accounts for 42% of the Google sentence compression dataset. As shown in Figure 2, long sentences …

Content truncated to 500 words