Improving Referee-Selection and Manuscript Evaluation

To maximize the chance that manuscripts will be evaluated by the most competent referees, four systematic procedures arc proposed. One expands the pool of potential referees. Another selects a set of people from that pool to evaluate a given manuscript. The third solicits from them judgments along five scales. The fourth procedure combines these judgments into a form of consensus to help the editor make a decision. The five scales involve measures of relevance to the journal’s objectives, significance of the problem, validity of the results, novelty and clarity. Limited experiments with field trials over the past five years with the Journal of the Association for Computing Machines show that implementations of some aspects of these procedures help to improve the quality of accepted papers. The quality of a paper is interpreted primarily as a likelihood of its being appropriately cited. Analysis of these algorithms leads to interesting research problems, such as the possibility of non-subjective measures of quality by the use of modified co-citation counts. The use of computer conferencing to aid communication among referees, and among them, the author and editors, is analyzed for its potentialities and constraints.