Article type
Year
Abstract
Introduction:
Objectives: To examine the methods used for assessing the quality of component studies in meta-analyses of randomised controlled trials published in English-language medical journals.
Methods: A handsearch 1993-1997 of four general medicine journals (Ann Intern Med, BMJ, Lancet, JAMA) and four specialist journals (Am J Cardiol, Cancer, Circulation, Obstet Gynot) identified 135 meta-analyses. We used a standardised questionnaire to extract relevant information.
Results: Authors of 47 (34.8%) meta-analyses assessed some dimension of quality of primary studies, with little evidence of a trend in the frequency of quality assessment over time. Beneficial and statistically significant results for primary outcomes were reported in about 70%, independent of whether the quality of primary studies was assessed or not.Meta-analyses sponsored by the pharmaceutical industry were less likely (2 out of 18, 11.1 %) to have assessed the quality of component studies than meta-analyses with other or unknown sources of funding (44 out of 117 37.6%, p=0.03). Conversely, articles which referred to the Cochrane Collaboration were more likely to include an assessment of study quality (11 out of 20, 55%) than articles that did not (35 out of 115, 30.4%, p=0.03). A total of 33 different scales and checklists were used. For 26 meta-analyses (55.3%) authors developed their own instrument. Study quality was incorporated in the analysis in 26 articles (55.3%). The most common method of incorporating quality data was to exclude trials deemed to be of poor quality (table).
Instrument used for quality assessment
Checklist 20 (42.6%)
Scale 29 (61.7%)
Methodological domain covered
Concealment of treatment allocation 31 (66.0%)
Blinding of outcome assessment 29 (61.7%)
Withdrawals and dropouts 27 (57.4%0
Use of quality data in analysis
Any use of quality data 26 (55.3%)
Exclusion of trials of low quality 15 (31.9%)
Analysis stratified by quality 4 (8.5%)
Trials weighted by quality 3 (6.4%)
Quality included in regression analysis 3(6.4%)
Discussion: Our findings indicate that there is widespread uncertainty about how best to assess the quality of primary studies. Methodological research is needed to clarify this important issue.
Objectives: To examine the methods used for assessing the quality of component studies in meta-analyses of randomised controlled trials published in English-language medical journals.
Methods: A handsearch 1993-1997 of four general medicine journals (Ann Intern Med, BMJ, Lancet, JAMA) and four specialist journals (Am J Cardiol, Cancer, Circulation, Obstet Gynot) identified 135 meta-analyses. We used a standardised questionnaire to extract relevant information.
Results: Authors of 47 (34.8%) meta-analyses assessed some dimension of quality of primary studies, with little evidence of a trend in the frequency of quality assessment over time. Beneficial and statistically significant results for primary outcomes were reported in about 70%, independent of whether the quality of primary studies was assessed or not.Meta-analyses sponsored by the pharmaceutical industry were less likely (2 out of 18, 11.1 %) to have assessed the quality of component studies than meta-analyses with other or unknown sources of funding (44 out of 117 37.6%, p=0.03). Conversely, articles which referred to the Cochrane Collaboration were more likely to include an assessment of study quality (11 out of 20, 55%) than articles that did not (35 out of 115, 30.4%, p=0.03). A total of 33 different scales and checklists were used. For 26 meta-analyses (55.3%) authors developed their own instrument. Study quality was incorporated in the analysis in 26 articles (55.3%). The most common method of incorporating quality data was to exclude trials deemed to be of poor quality (table).
Instrument used for quality assessment
Checklist 20 (42.6%)
Scale 29 (61.7%)
Methodological domain covered
Concealment of treatment allocation 31 (66.0%)
Blinding of outcome assessment 29 (61.7%)
Withdrawals and dropouts 27 (57.4%0
Use of quality data in analysis
Any use of quality data 26 (55.3%)
Exclusion of trials of low quality 15 (31.9%)
Analysis stratified by quality 4 (8.5%)
Trials weighted by quality 3 (6.4%)
Quality included in regression analysis 3(6.4%)
Discussion: Our findings indicate that there is widespread uncertainty about how best to assess the quality of primary studies. Methodological research is needed to clarify this important issue.