E-TIPSY: Search Query Corpus Annotated with Entities, Term Importance, POS Tags, and Syntactic Parses

E-TIPSY: Search Query Corpus Annotated with Entities, Term Importance, POS Tags, and Syntactic Parses Yuval Marton and Kristina Toutanova Microsoft Corporation Redmond, WA, USA {yumarton, kristout}@microsoft.com Abstract We present E-TIPSY, a search query corpus annotated with named Entities, Term Importance, POS tags, and SYntactic parses. This corpus contains crowdsourced (gold) annotations of the three most important terms in each query. In addition, it contains automatically produced annotations of named entities, part-of-speech tags, and syntactic parses for the same queries. This corpus comes in two formats: (1) Sober Subset: annotations that two or more crowd workers agreed upon, and (2) Full Glass: all annotations. We analyze the strikingly low correlation between term importance and syntactic headedness, which invites research into effective ways of combining these different signals. Our corpus can serve as a benchmark for term importance methods aimed at improving search engine quality and as an initial step toward developing a dataset of gold linguistic analysis of web search queries. In addition, it can be used as a basis for linguistic inquiries into the kind of expressions used in search. Keywords: term importance, query analysis, query parsing 1 Introduction Search engines can easily serve highly relevant documents for common ("head") queries, simply by using past user click data for these queries. But there is no sufficient click data, if any at all, for rare ("tail") queries and new queries. For such queries search engines have to resort to more sophisticated methods that analyze queries based on the sequence of terms they contain. Term weighting is used to prioritize relevant documents. Weights are assigned by the search engine, with simple manual rules or sophisticated machine learning algorithms. Deeper natural language processing (NLP) analyses such as syntactic parsing can also help determine term importance and estimate document relevance. We present a corpus consisting of tail queries issued to the Bing search engine,1 together with manual annotation of term importance. In addition, our corpus contains annotation of named entities, part-of-speech (POS) tags, and syntactic analyses, generated using state-of-the-art NLP models. Along with the above, we include manual annotation of head-words for a small subset of the query set. Based on this manual annotation, we estimate the automatic parser accuracy and estimate the extent of correspondence between syntactic heads and most important terms. A linguistically annotated corpus of tail queries is needed because queries are linguistically different from typical documents. This difference results in low quality of current NLP tools’ output. We argue this quality can be improved with a resource such as ours, as has been the case with other task-specific annotations. Our corpus can be used as a benchmark for new models of 1 Bing.com term importance in web search. It can also be used to study the impact of linguistic analysis on this task-driven problem, and to understand and improve the performance of NLP models in the domain of search queries. 2 Data Collection and Annotation 2.1 Query Set We used English queries issued to the Bing search engine in the …

Content truncated to 500 words