Feedback on Cochrane Reviews: how much is there and what happens to it?

Hilton J1, Stewart G2, Abaring T3, Cracknell J4
1Cochrane Editorial Unit, United Kingdom, 2John Wiley & Sons Ltd, United Kingdom, 3John Wiley & Sons Inc, USA, 4Cochrane Anaesthesia Group, Denmark

Background:

Users of The Cochrane Library are able to submit comments on Cochrane Reviews and Protocols, enabling review authors and editorial teams to amend or update reviews when they are made aware of new information, errors, or areas of confusion. The main aim of soliciting comments is to improve quality of published reviews, but we know little about the fate and impact of recently submitted comments.

Objectives:

To explore how the number and characteristics of comments on Cochrane Reviews and Protocols has changed over time, to assess how many comments are published, what their characteristics are, how many are responded to, how long this process takes, and whether the review is amended or updated as a result of the comment.

Methods:

We looked at all comments submitted via The Cochrane Lbrary website or the Ovid platforms during 2013. We compared the origin, number and Cochrane Review Group (CRG) coverage of the comments with data from previous years. We intend to continue gathering data and will present information on comments submitted during early 2014. We will also analyse a sample of submitted comments to assess the topics addressed by the comments, whether and when they were published, with or without a response from the author. We will also assess whether reviews were amended or updated in light of the comments, and we will look for other impacts of the comments.

Results:

During 2013, the Wiley team processed 158 submitted comments, 139 of which were assessed as relevant and passed to CRGs. This represents a growth of 62% compared with 2012. Forty-seven of the 53 CRGs received at least one comment, and two CRGs received 10 or more comments each. More recent data will be presented alongside an analysis of the fate of the submitted comments.

Conclusions:

The Cochrane Library comments system is a regularly used forum for readers of Cochrane Reviews and Protocols to participate in post-publcation peer review. More information on the fate and impact of comments will help us to understand the degree to which Cochrane Reviews are influenced by comments, as well as enabling us to ensure that the comments system is working effectively.