Article type
Year
Abstract
Background: Previous studies have shown considerable variability of the reliability of different raters to achieve consistant results when undertaken quality assessments.
Objectives: To determine the agreement between post-graduate health sciences students using commonly used critical appraisal techniques on a common set of literature.
Methods: 19 students undertaking a Masters level systematic review module scrutinised 10 clinical trials chosen at random from the specialised clinical trial register of the Cochrane Wounds Group. Participants rated each paper using both the Jadad score and the Cochrane Collaboration grading of allocation concealment (adequate, inadequate or unclear). Intraclass cluster coefficients (ICC) and chance corrected agreement (CohenÂs kappa) between raters for each paper using each grading system was then calculated.
Results: Analysis of agreement of the Jadad scores resulted in an intraclass correlation coefficient of 0.50 (95% CI = 0.25 to 0.75), meaning that half the variation in Jadad scores was due to observer variation and half due to true differences between the papers.
Kappa scores for each question in the Jadad tool were as follows:
Question kappa
Q1 Was the study described as randomised? 0.81
Q2 Was the study described as double blind? 0.70
Q3 Was there a description of withdrawals
and drop outs? 0.42
Q4 If randomisation appropriate 0.33
Q5 If blinding appropriate 0.25
Q6 Randomisation inappropriate 0.11
Q7 Blinding inappropriate 0.00
ICC and kappa scores for the Cochrane assessment of allocation concealment were as follows: ICC = 0.29 (C.I. 0.06-0.51) meaning that 29% of variability is explained by true differences between the study population and the further 71% of variability due to oberver variation. The unweighted kappa = 0.18 further supported that there was considerable variation between the responses of the students to the question asked. Further results for another cohort of students will be added to the results before the Colloquium.
Conclusions: Considerable variability exists in the responses of postgraduate students recently taught critical appraisal using these assessment tools. These results are consistant with results of previous studies that have looked at the Jadad scale and provide first time evidence that similar issues may be evident in Cochrane quality assessments.
Objectives: To determine the agreement between post-graduate health sciences students using commonly used critical appraisal techniques on a common set of literature.
Methods: 19 students undertaking a Masters level systematic review module scrutinised 10 clinical trials chosen at random from the specialised clinical trial register of the Cochrane Wounds Group. Participants rated each paper using both the Jadad score and the Cochrane Collaboration grading of allocation concealment (adequate, inadequate or unclear). Intraclass cluster coefficients (ICC) and chance corrected agreement (CohenÂs kappa) between raters for each paper using each grading system was then calculated.
Results: Analysis of agreement of the Jadad scores resulted in an intraclass correlation coefficient of 0.50 (95% CI = 0.25 to 0.75), meaning that half the variation in Jadad scores was due to observer variation and half due to true differences between the papers.
Kappa scores for each question in the Jadad tool were as follows:
Question kappa
Q1 Was the study described as randomised? 0.81
Q2 Was the study described as double blind? 0.70
Q3 Was there a description of withdrawals
and drop outs? 0.42
Q4 If randomisation appropriate 0.33
Q5 If blinding appropriate 0.25
Q6 Randomisation inappropriate 0.11
Q7 Blinding inappropriate 0.00
ICC and kappa scores for the Cochrane assessment of allocation concealment were as follows: ICC = 0.29 (C.I. 0.06-0.51) meaning that 29% of variability is explained by true differences between the study population and the further 71% of variability due to oberver variation. The unweighted kappa = 0.18 further supported that there was considerable variation between the responses of the students to the question asked. Further results for another cohort of students will be added to the results before the Colloquium.
Conclusions: Considerable variability exists in the responses of postgraduate students recently taught critical appraisal using these assessment tools. These results are consistant with results of previous studies that have looked at the Jadad scale and provide first time evidence that similar issues may be evident in Cochrane quality assessments.