Article type
Year
Abstract
Background: Jüni et al1 evaluated 25 Quality Assessment Scales using 17 trials comparing heparins for thromboprophylaxis in general surgery, and found that the type of scale used to assess trial quality dramatically influenced the interpretation of the meta-analysis. They noted that "strictly speaking, the results are only applicable to the 17 trials examined". Previous studies2-4 have shown that about one third of large trials do not agree with a meta-analysis of smaller trials. Large trials have been advocated as a 'reference standard' for the evaluation of effectiveness of interventions2, although this view has been challenged.3
Objectives: To extend the study of Jüni to more outcomes and a larger number of quality scores, and to explore the validity of adjusting the meta-analysis of the small studies for quality, using at least one large study as a reference standard.
Methods: An observational study of meta-analyses from Issue 1, 2001 of The Cochrane Library. Selected meta-analyses had to have binary outcomes and include more than 10 studies, at least one of which randomized more than 500 people into each group.
Results: Sixty-five meta-analyses using binary outcomes from 18 systematic reviews were scored using 43 quality scores. No score was better at dividing the studies into low and high quality, and none of the scores was better over 65 meta-analyses in making the result closer to the reference standard.
Conclusions: None of the quality scores found appeared to measure quality validly over a wide range of studies. It is a mistake to assign meaning to the result of a quality score. Large trials may not be a very good reference standard.
References
1. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054-60.
2. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large
randomized, controlled trials. New England Journal of Medicine 1997;337:536-42.
3. Cappelleri JC, Ioannidis JP, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, Lau J. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996;276:1332-8.
4. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomized controlled trials. Lancet 1995;345:772-6.
Percentage of the 65 meta-analyses with different levels of agreement between mega-trials and 'high' and 'low' quality smaller studies for each of the quality scores.
Objectives: To extend the study of Jüni to more outcomes and a larger number of quality scores, and to explore the validity of adjusting the meta-analysis of the small studies for quality, using at least one large study as a reference standard.
Methods: An observational study of meta-analyses from Issue 1, 2001 of The Cochrane Library. Selected meta-analyses had to have binary outcomes and include more than 10 studies, at least one of which randomized more than 500 people into each group.
Results: Sixty-five meta-analyses using binary outcomes from 18 systematic reviews were scored using 43 quality scores. No score was better at dividing the studies into low and high quality, and none of the scores was better over 65 meta-analyses in making the result closer to the reference standard.
Conclusions: None of the quality scores found appeared to measure quality validly over a wide range of studies. It is a mistake to assign meaning to the result of a quality score. Large trials may not be a very good reference standard.
References
1. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054-60.
2. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large
randomized, controlled trials. New England Journal of Medicine 1997;337:536-42.
3. Cappelleri JC, Ioannidis JP, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, Lau J. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996;276:1332-8.
4. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomized controlled trials. Lancet 1995;345:772-6.
Percentage of the 65 meta-analyses with different levels of agreement between mega-trials and 'high' and 'low' quality smaller studies for each of the quality scores.