Incremental Beam Manipulation for Natural Language Generation
42489Authors
nan
Abstract
AbstractThe performance of natural language generation systems has improved substantially with modern neural networks. At test time they typically employ beam search to avoid locally optimal but globally suboptimal predictions. However, due to model errors, a larger beam size can lead to deteriorating performance according to the evaluation metric. For this reason, it is common to rerank the output of beam search, but this relies on beam search to produce a good set of hypotheses, which limits the potential gains. Other alternatives to beam search require changes to the training of the model, which restricts their applicability compared to beam search. This paper proposes incremental beam manipulation, i.e. reranking the hypotheses in the beam during decoding instead of only at the end. This way, hypotheses that are unlikely to lead to a good final output are discarded, and in their place hypotheses that would have been ignored will be considered instead. Applying incremental beam manipulation leads to an improvement of 1.93 and 5.82 BLEU points over vanilla beam search for the test sets of the E2E and WebNLG challenges respectively. The proposed method also outperformed a strong reranker by 1.04 BLEU points on the E2E challenge, while being on par with it on the WebNLG dataset.
Keywords
Full Content
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 270–280 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 270 CD2CR: Co-reference Resolution Across Documents and Domains James Ravenscroft1,2,3 Arie Cattan4 Amanda Clare5 Ido Dagan4 Maria Liakata1,2 1: University of Warwick, United Kingdom 2: Alan Turing Institute, United Kingdom 3: Filament AI, United Kingdom 4: Bar-Ilan University, Israel 5: Aberystwyth University, United Kingdom {j.ravenscroft|m.liakata}@warwick.ac.uk {arie.cattan|ido.k.dagan}@gmail.com afc@aber.ac.uk Abstract Cross-document co-reference resolution (CDCR) is the task of identifying and linking mentions to entities and concepts across many text documents. Current state-of-the-art models for this task assume that all documents are of the same type (e.g. news articles) or fall under the same theme. However, it is also desirable to perform CDCR across different domains (type or theme). A particular use case we focus on in this paper is the resolution of entities mentioned across scientific work and newspaper articles that discuss them. Identifying the same entities and correspond- ing concepts in both scientific articles and news can help scientists understand how their work is represented in mainstream media. We propose a new task and English language dataset for cross-document cross-domain co-reference resolution (CD2CR). The task aims to identify links between entities across heterogeneous document types. We show that in this cross-domain, cross-document setting, existing CDCR models do not perform well and we provide a baseline model that outperforms current state-of-the-art CDCR models on CD2CR. Our data set, annotation tool and guidelines as well as our model for cross-document cross-domain co-reference are all supplied as open access open source resources. 1 Introduction Cross-document co-reference resolution (CDCR) is the task of recognising when multiple docu- ments mention and refer to the same real-world entity or concept. CDCR is a useful NLP pro- cess that has many downstream applications. For example, CDCR carried out on separate news ar- ticles that refer to the same politician can facili- tate inter-document sentence alignment required for stance detection and natural language inference models. Furthermore, CDCR can improve informa- tion retrieval and multi-document summarisation by grouping documents based on the entities that are mentioned within them. Recent CDCR work (Dutta and Weikum, 2015; Barhom et al., 2019; Cattan et al., 2020) has pri- marily focused on resolution of entity mentions across news articles. Despite differences in tone and political alignment, most news articles are rel- atively similar in terms of grammatical and lexi- cal structure. Work based on modern transformer networks such as BERT (Devlin et al., 2019) and ElMo (Peters et al., 2018) have been pre-trained on large news corpora and are therefore well suited to news-based CDCR (Barhom et al., 2019). However, there are cases where CDCR across documents from different domains (i.e. that dif- fer much more significantly in style, vocabulary and structure) is useful. One such example is the task of resolving references to concepts across sci- entific papers and related news articles. This can help scientists understand how their work is being presented to the public …
Content truncated to 500 words
Labeled Text
AbstractThe performance of natural language generation systems has improved substantially with modern neural networks. At test time they typically employ beam search to avoid locally optimal but globally suboptimal predictions. However, due to model errors, a larger beam size can lead to deteriorating performance according to the evaluation metric. For this reason, it is common to rerank the output of beam search, but this relies on beam search to produce a good set of hypotheses, which limits the potential gains. Other alternatives to beam search require changes to the training of the model, which restricts their applicability compared to beam search. This paper proposes incremental beam manipulation, i.e. reranking the hypotheses in the beam during decoding instead of only at the end. This way, hypotheses that are unlikely to lead to a good final output are discarded, and in their place hypotheses that would have been ignored will be considered instead. Applying incremental beam manipulation leads to an improvement of 1.93 and 5.82 BLEU points over vanilla beam search for the test sets of the E2E and WebNLG challenges respectively. The proposed method also outperformed a strong reranker by 1.04 BLEU points on the E2E challenge, while being on par with it on the WebNLG dataset.
Publication Details
- Paper ID:
- 42489
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan