Article type
Year
Abstract
Background: Biomedical journals and the Cochrane Library strive to publish high quality research to improve public health. To reach this goal, editors incorporate expert clinical and statistical peer review. Yet, little is known about how peer reviewer comments are considered and how much influence they have on editors' decisions to accept or reject a manuscript.
Objectives: To assess whether peer reviewer recommendations agree with editorial decisions of acceptance or rejection and whether peer reviewers agree among themselves for a particular manuscript.
Methods: Prospective study of three major biomedical journals. Approximately 1,186 manuscripts of original research were submitted for publication January 2003 - April 2003. All corresponding peer review and statistical comments were collected. Manuscript acceptance or rejection was determined by tracking the original submission through each journalÂs editorial process. Peer reviewer recommendations were categorized as accept, accept minor revisions, major revisions, and reject. To evaluate associations of peer reviewer recommendations on editorial decisions to accept or reject, we used binary logistic regression. Agreement between two or more reviewers for a particular manuscript in the four categories was assessed using the kappa statistic.
Results: Among the three journals, a total of 53 accepted and 164 rejected manuscripts evaluated by one or more peer reviewers were available for analysis. There were significant associations between editors' decision to accept and peer reviewers' recommendations to accept (P<0.001; OR 32.30; 95% CI 6.09 - 171.25) or accept with minor revisions (P=0.001; OR 7.99; 95% CI 2.27 - 28.12). Recommendation for major revisions (P=0.08; OR 3.29; 95% CI 0.88 - 12.32) was not statistically different from rejection (referent category). Increasing the number of peer reviewers was inversely related to acceptance (P<0.001; OR 0.08; 95% CI 0.02 - 0.29). Concordance between acceptance and peer reviewer recommendations was 91.5% (Goodman-Kruskal's Gamma =0.87). Overall agreement between peer reviewers was poor and worsened as the number of reviewers increased (kappa = 0.13, -0.73, and -1.92 for 2, 3, and 4 reviewers, respectively).
Conclusions: Peer reviewers and editors agree on manuscript acceptance or rejection, suggesting that peer reviewers influence the editorial decision-making process. Additionally, editors face difficulty in resolving conflicting reviews.
Objectives: To assess whether peer reviewer recommendations agree with editorial decisions of acceptance or rejection and whether peer reviewers agree among themselves for a particular manuscript.
Methods: Prospective study of three major biomedical journals. Approximately 1,186 manuscripts of original research were submitted for publication January 2003 - April 2003. All corresponding peer review and statistical comments were collected. Manuscript acceptance or rejection was determined by tracking the original submission through each journalÂs editorial process. Peer reviewer recommendations were categorized as accept, accept minor revisions, major revisions, and reject. To evaluate associations of peer reviewer recommendations on editorial decisions to accept or reject, we used binary logistic regression. Agreement between two or more reviewers for a particular manuscript in the four categories was assessed using the kappa statistic.
Results: Among the three journals, a total of 53 accepted and 164 rejected manuscripts evaluated by one or more peer reviewers were available for analysis. There were significant associations between editors' decision to accept and peer reviewers' recommendations to accept (P<0.001; OR 32.30; 95% CI 6.09 - 171.25) or accept with minor revisions (P=0.001; OR 7.99; 95% CI 2.27 - 28.12). Recommendation for major revisions (P=0.08; OR 3.29; 95% CI 0.88 - 12.32) was not statistically different from rejection (referent category). Increasing the number of peer reviewers was inversely related to acceptance (P<0.001; OR 0.08; 95% CI 0.02 - 0.29). Concordance between acceptance and peer reviewer recommendations was 91.5% (Goodman-Kruskal's Gamma =0.87). Overall agreement between peer reviewers was poor and worsened as the number of reviewers increased (kappa = 0.13, -0.73, and -1.92 for 2, 3, and 4 reviewers, respectively).
Conclusions: Peer reviewers and editors agree on manuscript acceptance or rejection, suggesting that peer reviewers influence the editorial decision-making process. Additionally, editors face difficulty in resolving conflicting reviews.