Article type
Year
Abstract
Background: The Cochrane Library relies on the regular update of reviews following new information and submitted criticisms in order to present the best, most carefully analyzed information. The role of criticisms as a means for updating reviews remains under-researched. The BMJ regularly publishes Cochrane reviews and their criticisms both electronically and in print. Thus, the BMJ can serve as a comparison for ascertaining and understanding the efficacy and challenges facing post-publication peer review in the Cochrane Library.
Objectives: To compare the submission rate of criticisms, the content of these criticisms, and author responses to these criticisms published in the BMJ, electronically and in print, with the submission rate, content of criticisms, and author responses submitted to the Cochrane Library.
Methods: Criticisms of Cochrane reviews published in the BMJ were found by 1) performing a MEDLINE search for every BMJ article published between January, 1996 and January, 2000 containing the words "systematic review" or "meta-analysis" in the title, 2) selecting only those articles that specifically stated that they can also be found in the Cochrane Library, and 3) collecting all published criticisms, author responses, and errata related to these articles. All criticisms and author responses received through the Criticism Management System of the Cochrane Library between January, 1996 and January, 2000 were included in this study. Every criticism from both the BMJ and the Cochrane Library were coded independently by RS and MO and counted.
Results: Submission Rate. We found 19 Cochrane reviews published in the BMJ that received a total of 83 criticisms (48 published electronically, 35 in print). However, 18 of the criticisms were published twice (one electronically, the other in print), resulting in a final count of 65 unique criticisms. The average submission rate in the BMJ was 3.4 (65/19) criticisms per review. The Cochrane Library received 171 criticisms for 716 reviews contained in 2000, Issue 1 for an average submission rate of 0.2 criticisms per review. Content of Criticisms. Among the criticisms submitted to the BMJ, 29% (19/65) criticized the conclusions for being unsubstantiated or misleading, 25% presented data to support the criticisms, and 20% (13/65) expressed concern over the heterogeneity of the studies included in the meta-analysis. Nineteen percent of criticisms (32/171) submitted to the Cochrane Library disagreed with or questioned the rationale for including or excluding certain studies in the analysis. Another 16% (27/171) requested more information about these studies. Authors who submitted criticisms to the Cochrane Library also expressed concern that conclusions were misleading or unsubstantiated in 13% (23/171) of the criticisms. Author Response Rate. The BMJ published 10 unique author responses (one published twice), thus 15% (10/ 65) of criticisms received responses. Similarly, 15% (26/171) of criticisms submitted to the Cochrane Library received author responses, resulting in nineteen reviews changing as a result. Only one correction was published for a Cochrane review in the BMJ, however the correction involved a typographical error.
Discussion: Criticisms submitted to the BMJ tended to emphasize how the review would impact research or clinical medicine and became part of an on-going discussion while criticisms submitted to the Cochrane Library focused on methodology. The speed with which criticisms submitted to the BMJ are posted on the web probably contributes to the perceived impact and stimulates dialogue. The Cochrane Library lacks this speed. However, the fact that authors can change their reviews as a result of criticisms is a unique strength that should be cultivated. The Cochrane Collaboration needs to focus on ways in which to improve the turnaround time for publishing criticisms and could consider using an electronic bulletin board to stimulate discussion.
Objectives: To compare the submission rate of criticisms, the content of these criticisms, and author responses to these criticisms published in the BMJ, electronically and in print, with the submission rate, content of criticisms, and author responses submitted to the Cochrane Library.
Methods: Criticisms of Cochrane reviews published in the BMJ were found by 1) performing a MEDLINE search for every BMJ article published between January, 1996 and January, 2000 containing the words "systematic review" or "meta-analysis" in the title, 2) selecting only those articles that specifically stated that they can also be found in the Cochrane Library, and 3) collecting all published criticisms, author responses, and errata related to these articles. All criticisms and author responses received through the Criticism Management System of the Cochrane Library between January, 1996 and January, 2000 were included in this study. Every criticism from both the BMJ and the Cochrane Library were coded independently by RS and MO and counted.
Results: Submission Rate. We found 19 Cochrane reviews published in the BMJ that received a total of 83 criticisms (48 published electronically, 35 in print). However, 18 of the criticisms were published twice (one electronically, the other in print), resulting in a final count of 65 unique criticisms. The average submission rate in the BMJ was 3.4 (65/19) criticisms per review. The Cochrane Library received 171 criticisms for 716 reviews contained in 2000, Issue 1 for an average submission rate of 0.2 criticisms per review. Content of Criticisms. Among the criticisms submitted to the BMJ, 29% (19/65) criticized the conclusions for being unsubstantiated or misleading, 25% presented data to support the criticisms, and 20% (13/65) expressed concern over the heterogeneity of the studies included in the meta-analysis. Nineteen percent of criticisms (32/171) submitted to the Cochrane Library disagreed with or questioned the rationale for including or excluding certain studies in the analysis. Another 16% (27/171) requested more information about these studies. Authors who submitted criticisms to the Cochrane Library also expressed concern that conclusions were misleading or unsubstantiated in 13% (23/171) of the criticisms. Author Response Rate. The BMJ published 10 unique author responses (one published twice), thus 15% (10/ 65) of criticisms received responses. Similarly, 15% (26/171) of criticisms submitted to the Cochrane Library received author responses, resulting in nineteen reviews changing as a result. Only one correction was published for a Cochrane review in the BMJ, however the correction involved a typographical error.
Discussion: Criticisms submitted to the BMJ tended to emphasize how the review would impact research or clinical medicine and became part of an on-going discussion while criticisms submitted to the Cochrane Library focused on methodology. The speed with which criticisms submitted to the BMJ are posted on the web probably contributes to the perceived impact and stimulates dialogue. The Cochrane Library lacks this speed. However, the fact that authors can change their reviews as a result of criticisms is a unique strength that should be cultivated. The Cochrane Collaboration needs to focus on ways in which to improve the turnaround time for publishing criticisms and could consider using an electronic bulletin board to stimulate discussion.