Quality in the review process

Referees often disagree as to whether to accept or reject a manuscript for publication. Recent investigations have often focused on the study of standard rating forms that ask for appraisal of specified attributes of a manuscript. Scott asked reviewers to rate, on seven criteria, manuscripts submitted to the Journal of Personality and Social Psychology. There was some agreement on most of the criteria; correlations were often significant but never large. Importance of the problem and adequacy of design and analysis seemed to affect evaluation most strongly. Whether the use of such a rating form will lead to greater reliability of referees´ recommendations is an open question. There are many reasons why referees disagree, of course, and research in this area should be broadly based. The decision-making process of the editor is also very complex and should be the focus of empirical investigation.