Article type
Year
Abstract
Background: Scientific Review Committee (SRC) members mostly use qualitative methods for reaching decisions (approval or disapproval) on study protocols. Checklists for reporting scientific studies have helped improve transparency and presentation of studies in the public reporting of research findings. While the use of checklists has become prevalent in the arena of publishing, their role in the SRC evaluation and decision-making process has not been assessed.
Objective: To assess the efficacy of the SRC approval process using an objective checklist in addition to the current procedures versus standard operating procedures (SOP) alone for making a decision on approval or disapproval of scientific study protocols.
Methods: Two committee members were randomly assigned to review each protocol involving human subjects using the checklist plus the SOP or SOP alone. Checklists (e.g. CONSORT, STROBE etc.) recommended by scientific journals were matched to study design. The primary outcome was approval rate for checklist(s) versus no checklist(s) group. Differences in disapproval rates between the two groups were assessed using a Chi2 test. The agreement between reviewers (checklist versus no checklist) was assessed using the Kappa statistic.
Results: A total of 28 study protocols were reviewed in duplicate by 56 reviewers (10 participants were excluded due to incorrect review). Characteristics of reviewers in compared groups were similar (see table 1). The relative risk reduction in disapproval was 33% with use of checklist plus SOP compared with SOP alone (absolute risk reduction = 25%; relative risk = 0.67; 95% CI 0.44 to 1.03). The agreement between two reviewers (checklist versus no checklist) assessing the same protocol was poor, resulting in kappa of 0.07 (95% CI 0 to 0.39).
Conclusion: Use of checklist along with SOP results in lower disapproval compared with the SOP only. The lack of statistical significance is likely due to small sample size. Introduction of checklists in the study design phase ensures better methodological quality of study protocols and assists SRC members remove the bottleneck in approvals and serve as facilitators of research.
Objective: To assess the efficacy of the SRC approval process using an objective checklist in addition to the current procedures versus standard operating procedures (SOP) alone for making a decision on approval or disapproval of scientific study protocols.
Methods: Two committee members were randomly assigned to review each protocol involving human subjects using the checklist plus the SOP or SOP alone. Checklists (e.g. CONSORT, STROBE etc.) recommended by scientific journals were matched to study design. The primary outcome was approval rate for checklist(s) versus no checklist(s) group. Differences in disapproval rates between the two groups were assessed using a Chi2 test. The agreement between reviewers (checklist versus no checklist) was assessed using the Kappa statistic.
Results: A total of 28 study protocols were reviewed in duplicate by 56 reviewers (10 participants were excluded due to incorrect review). Characteristics of reviewers in compared groups were similar (see table 1). The relative risk reduction in disapproval was 33% with use of checklist plus SOP compared with SOP alone (absolute risk reduction = 25%; relative risk = 0.67; 95% CI 0.44 to 1.03). The agreement between two reviewers (checklist versus no checklist) assessing the same protocol was poor, resulting in kappa of 0.07 (95% CI 0 to 0.39).
Conclusion: Use of checklist along with SOP results in lower disapproval compared with the SOP only. The lack of statistical significance is likely due to small sample size. Introduction of checklists in the study design phase ensures better methodological quality of study protocols and assists SRC members remove the bottleneck in approvals and serve as facilitators of research.