Taking the load off the conference chairs-towards a digital paper-routing assistant

Taking the load off the conference chairs: towards a digital paper-routing assistant David Yarowsky and Radu Florian Computer Science Department and Center for Language and Speech Processing Johns Hopkins University Baltimore, Maryland 21218 {yarowsky, rflorian}@cs.jhu.edu Abstract This paper describes and extensively evaluates a sys- tem for the automatic routing of submitted papers to reviewers and area committees, without the need for any human annotation from the reviewers or the pro- gram chair. Routing is based on a profile of previous writings obtainable on-line for the reviewer pool, a generally stable and reusable resource that requires no manual adaptation for new submission streams. The paper explores a wide set of variations and ex- tensions on the core model, and achieves system ac- curacy approaching that of several human judges on the same task. 1 Introduction and Problem Statement Routing submitted papers, abstracts or grant pro- posals to qualified reviewers is a central task of the academic enterprise, and a remarkably difficult one. Typically it is conducted under significant time pres- sure in a conference reviewing cycle. As the number of submissions and size of the reviewer pool grows, it becomes increasingly difficult for a conference chair to be familiar with the different expertise of all mem- bers of the program committee. It is also difficult for one person to master the subtleties of fine sub- ject area distinctions as topic diversity in a confer- ence becomes large. For these reasons, conferences such as ACL (the Association for Computational Linguistics) often use a hierarchical program com- mittee structure, where submitted papers are first routed to area committees, and then more special- ized area chairs have the task of assigning papers to individual reviewers in the committee. However, in a diverse and multidisciplinary field such as natu- ral language processing, it is often difficult to define clear cut committee descriptions and the program chair still must be cognizant of the detailed exper- tise of the area committee members in order to route atypical or multidisciplinary papers to committees with the most appropriate pool of reviewers. The low inter-rater consistency results shown in Table 12 indicate that humans find even area committee routing to be a difficult task. The following paper focuses on a range of auto- mated solutions to this task of routing papers to their most appropriate area committee. It presents extensive empirical investigation and evaluation of a wide range of issues related to this task. Previous published research into the problem of automatic routing of conference paper submissions is surprisingly limited. Approaches to this task can be essentially broken down into four major strategies: The first strategy is keyword based. Authors are required to specify a list of topic/subtopic areas for their papers (often from a prespecified term list), and reviewers then complete a survey of their rela- tive level of expertise on this list of topics/subtopics. This approach is followed by AAAI conference re- viewing. It suffers from the problem that authors often have a difficult time selecting keywords to …

Content truncated to 500 words