Comparison of Diverse Decoding Methods from Conditional Language Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752–3762 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3752 Comparison of Diverse Decoding Methods from Conditional Language Models Daphne Ippolito⋆ Reno Kriz⋆ Maria Kustikova Jo˜ao Sedoc Chris Callison-Burch ⋆Authors contributed equally University of Pennsylvania {daphnei,rekriz,mkust,joao,ccb}@seas.upenn.edu Abstract While conditional language models have greatly improved in their ability to output high-quality natural language, many NLP ap- plications benefit from being able to generate a diverse set of candidate sequences. Diverse decoding strategies aim to, within a given- sized candidate list, cover as much of the space of high-quality outputs as possible, leading to improvements for tasks that re-rank and com- bine candidate outputs. Standard decoding methods, such as beam search, optimize for generating high likelihood sequences rather than diverse ones, though recent work has fo- cused on increasing diversity in these meth- ods. In this work, we perform an extensive survey of decoding-time strategies for generat- ing diverse outputs from conditional language models. We also show how diversity can be improved without sacrificing quality by over- sampling additional candidates, then filtering to the desired number. 1 Introduction Conditional neural language models, which train a neural net to map from one sequence to an- other, have had enormous success in natural lan- guage processing tasks such as machine transla- tion (Sutskever et al., 2014; Luong et al., 2015), text summarization (Nallapati et al., 2016), and di- alog systems (Vinyals and Le, 2015). These mod- els output a probability distribution over the next token in the output sequence given the input and the previously predicted tokens. Since comput- ing the overall most likely output sequence is in- tractable, early work in neural machine translation found that beam search is an effective strategy to heuristically sample sufficiently likely sequences from these probabilistic models (Sutskever et al., 2014). However, for more open-ended tasks, beam search is ill-suited to generating a set of diverse candidate sequences; this is because candidates Beam Search A bus is stopped at a bus stop. A bus is parked at a bus stop. A bus stopped at a bus stop in a city. A bus stopped at a bus stop at a bus stop. A bus that is parked in front of a building. Random Sampling A bus parked at a bus stop at a bus stop. There is a bus that is at the station. A man standing by a bus in a city. A bus pulling away from the train station. A bus stopped at a stop on the sunny day. Figure 1: An image with the top five captions from standard beam search and from random sampling. Note the latter set is more diverse but lower quality. outputted from a large-scale beam search often only differ by punctuation and minor morphologi- cal variations (Li and Jurafsky, 2016). The term “diversity” has been defined in a vari- ety of ways in the literature, with some using it as a synonym for …

Content truncated to 500 words