Making Information Seeking Easier: An Improved Pipeline for Conversational Search

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3971–3980 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3971 Making Information Seeking Easier: An Improved Pipeline for Conversational Search Vaibhav Kumar ∗ Language Technologies Institute Carnegie Mellon University vaibhav2@cs.cmu.edu Jamie Callan Language Technologies Institute Carnegie Mellon University callan@cs.cmu.edu Abstract This paper presents a highly effective pipeline for passage retrieval in a conversational search setting. The pipeline comprises of two com- ponents: Conversational Term Selection (CTS) and Multi-View Reranking (MVR). CTS is re- sponsible for performing the first-stage of pas- sage retrieval. Given an input question, it uses a BERT-based classifier (trained with weak su- pervision) to de-contextualize the input by se- lecting relevant terms from the dialog history. Using the question and the selected terms, it issues a query to a search engine to perform the first-stage of passage retrieval. On the other hand, MVR is responsible for contex- tualized passage reranking. It first constructs multiple views of the information need embed- ded within an input question. The views are based on the dialog history and the top doc- uments obtained in the first-stage of retrieval. It then uses each view to rerank passages us- ing BERT (fine-tuned for passage ranking). Fi- nally, MVR performs a fusion over the rank- ings produced by the individual views. Exper- iments show that the above combination im- proves first-state retrieval as well as the over- all accuracy in a reranking pipeline. On the key metric of NDCG@3, the proposed com- bination achieves a relative performance im- provement of 14.8% over the state-of-the-art baseline and is also able to surpass the Oracle. 1 Introduction The abilities of current conversational assistants (Alexa, Cortana etc.) to perform open-domain con- versational information seeking (CIS) functions are limited (Dalton et al., 2019). Thus, to encourage and support research on conversational informa- tion seeking, the TREC Conversational Assistance Track (CAsT) (Dalton et al., 2019) defined a model ∗ The author is now an Applied Scientist at Amazon Alexa AI. Alternatively, he can be contacted at vaibhav4595@gmail.com. Title: goat breeds Description: Interested in buying goats that implies interest in different breeds of goats and their use (milk, meat, and fur). Turn Utterance (Question) 1 What are the main breeds of goat? 2 Tell me about boer goats. 3 What breed is good for meat? 4 Are angora goats good for it? 5 What about boer goats? 6 What are pygmies used for? 7 What is the best for fiber production? 8 How long do Angora goats live? 9 Can you milk them? 10 How many can you have per acre? 11 Are they profitable? Table 1: An example of a training topic in CAsT. of conversational information seeking in which the conversation is a sequence of related passage rank- ing tasks, some of which require knowing the con- versational history. For example, the question “Can you milk them?” in Table 1 is not by itself sufficient to support effective retrieval; the conversational context is required. More formally, …

Content truncated to 500 words