Article type
Year
Abstract
Objective: A major issue in systematic reviews is what types of studies should be included or excluded from the review. The Cochrane Collaboration has traditionally viewed the RCT as the 'gold standard'. However, many Cochrane reviews are inconclusive because of insufficient RCTs while rejecting large numbers of non-randomised studies. This is particularly true in mental health, a domain where RCTs may be particularly problematic. Earlier research indicated that non-random studies produced larger treatment effect scores than RCTs in the same domain but subsequent research has cast doubt on this. An emerging, alternative view is that good quality non-randomised studies may be reliable surrogates for RCTs and systematic reviews could become more informative if reviewers knew when, and under what circumstances, non-randomised designs are acceptable for inclusion alongside RCTs. This pilot study tests a method to explore the issue further.
Method: Two reviews from the Cochrane Schizophrenia Group were selected and the results of the RCTs included in the review were compared with the outcomes of non-randomised control studies excluded from the reviews. Comparison was made between randomised and non-randomised studies and between high and low quality studies as measured by Downes and Black's quality checklist.
Results: It was only possible to identify three outcomes shared by both randomised and non-randomised studies for comparison. One outcome showed greater treatment effect in RCTs, another showed RCTs having a smaller treatment effect and a third showed no difference. There was a consistent finding, though varying in magnitude, that high quality studies showed smaller treatment effects than low quality studies.
Conclusions: These limited findings lend support to the view that the quality of the study has a greater impact on treatment effect size than randomisation alone and that randomisation should not be seen as a reliable proxy for overall quality. The problems and issues still to be resolved are discussed with recommendations for future research.
Method: Two reviews from the Cochrane Schizophrenia Group were selected and the results of the RCTs included in the review were compared with the outcomes of non-randomised control studies excluded from the reviews. Comparison was made between randomised and non-randomised studies and between high and low quality studies as measured by Downes and Black's quality checklist.
Results: It was only possible to identify three outcomes shared by both randomised and non-randomised studies for comparison. One outcome showed greater treatment effect in RCTs, another showed RCTs having a smaller treatment effect and a third showed no difference. There was a consistent finding, though varying in magnitude, that high quality studies showed smaller treatment effects than low quality studies.
Conclusions: These limited findings lend support to the view that the quality of the study has a greater impact on treatment effect size than randomisation alone and that randomisation should not be seen as a reliable proxy for overall quality. The problems and issues still to be resolved are discussed with recommendations for future research.