Article type
Year
Abstract
Introduction:
Objectives: Continual post-publication criticism and updating of reviews should help produce the best quality reviews. Thus, in April 1997 the Cochrane Collaboration implemented a Criticism Management System (CMS) that allowed readers to submit feedback electronically. In an attempt to increase the number of criticisms received and responded to, the original system was revamped and re-released in January 1999. Major changes included: a simplified interface that no longer requires users to structure their criticisms by specific sections of the review, the criticisms are now sent directly to Criticism Editors whereas previously the Criticism Editors had to retrieve them from a web site, and a summary of the criticism is now posted publicly on a "Current Comments and Criticisms" web page. Is there a relationship between these changes and the number of criticisms sent in as well as the number of criticisms responded to by the reviewers? To answer this question we assessed the number of criticisms being sent in and the number of criticisms being responded to by the reviewers before and after changes in the system. We also surveyed the Readers, the Criticism Editors and the Reviewers regarding their perceptions of the new CMS.
Methods: We counted the number of criticisms received and responded to via the old system and the new system. We also recorded the source and type of criticism. We surveyed by email: Readers (n=8), Criticism Editors (n=14), and Reviewers (n=14) who had used both the old and the new CMS.
Results: From April 1997 through October 1998 readers sent in 60 criticisms out of a total of 481 reviews, producing an average of 0.006 criticisms/per review/per month. Reviewers responded to 24 of these criticisms at an average of 0.79 a month. Utilizing the new CMS from January through April of 1999 readers sent in 33 criticisms out of a total of 522 reviews, producing an average of 0.015 criticisms/per review/per month. Reviewers responded to 3 of these criticisms at an average of 0.75 per month. If we exclude the 5 criticisms that have been sent in during March and April of 1999 as part of an internal Cochrane effort to provide criticism, the number of criticisms after changes to the CMS decreases to an average of 0.014 criticisms/per review/per month.
Discussion: There has been an increase in the number of criticisms being sent in, but not in the number of responses to them. Possible reasons for this result and user's perceptions of the new CMS will be discussed in the presentation.
Objectives: Continual post-publication criticism and updating of reviews should help produce the best quality reviews. Thus, in April 1997 the Cochrane Collaboration implemented a Criticism Management System (CMS) that allowed readers to submit feedback electronically. In an attempt to increase the number of criticisms received and responded to, the original system was revamped and re-released in January 1999. Major changes included: a simplified interface that no longer requires users to structure their criticisms by specific sections of the review, the criticisms are now sent directly to Criticism Editors whereas previously the Criticism Editors had to retrieve them from a web site, and a summary of the criticism is now posted publicly on a "Current Comments and Criticisms" web page. Is there a relationship between these changes and the number of criticisms sent in as well as the number of criticisms responded to by the reviewers? To answer this question we assessed the number of criticisms being sent in and the number of criticisms being responded to by the reviewers before and after changes in the system. We also surveyed the Readers, the Criticism Editors and the Reviewers regarding their perceptions of the new CMS.
Methods: We counted the number of criticisms received and responded to via the old system and the new system. We also recorded the source and type of criticism. We surveyed by email: Readers (n=8), Criticism Editors (n=14), and Reviewers (n=14) who had used both the old and the new CMS.
Results: From April 1997 through October 1998 readers sent in 60 criticisms out of a total of 481 reviews, producing an average of 0.006 criticisms/per review/per month. Reviewers responded to 24 of these criticisms at an average of 0.79 a month. Utilizing the new CMS from January through April of 1999 readers sent in 33 criticisms out of a total of 522 reviews, producing an average of 0.015 criticisms/per review/per month. Reviewers responded to 3 of these criticisms at an average of 0.75 per month. If we exclude the 5 criticisms that have been sent in during March and April of 1999 as part of an internal Cochrane effort to provide criticism, the number of criticisms after changes to the CMS decreases to an average of 0.014 criticisms/per review/per month.
Discussion: There has been an increase in the number of criticisms being sent in, but not in the number of responses to them. Possible reasons for this result and user's perceptions of the new CMS will be discussed in the presentation.