Article type
Year
Abstract
Objective: To analyse:- 1. criteria used to evaluate the methodological quality of primary studies and how they have been classified in A,B,C,D as required by RevMan. 2. how the methodological quality of primary studies has been incorporated in the review both in the methods and in the results section.
Methods: We checked the 19 reviews and 11 protocols published by the Group (CLib 1.2004) to answer the following questions: ? Which criteria were inspected for the quality assessment of studies? ? How were they used to classify studies in the A,B,C,D classes? ? How the quality assessment results were meant to be used in the review (i.e. for inclusion criteria, for sensitivity analysis) or only described? ? If intention to use results of quality assessment is actually adopted in the results section. The first three questions were applied both to reviews and protocols, the last one only to reviews.
Results: Criteria used for the quality assessment: ? Not specified: 3/30 (1%) ? Guideline of the Drug and Alcohol Group:8/30 (26.6%) ? Allocation concealment only: 12/30 (40%) ? Randomization method and allocation concealment: 2/30 (0.6) ? Randomization method, allocation concealment, and follow up: 2/30 (0,6%) ? Randomization method, allocation concealment, blinding, and follow up: 3/30 (1%) Criteria used to classify the studies in the A,B,C,D classes ? Not specified: 12/30 (40%) ? Allocation concealment: 14/30 (46.6%) ? Randomization method, allocation concealment: 1/30 (0.3%) ? Randomization method, allocation concealment, follow up: 2/30 (0,6%) ? Randomization method, allocation concealment, blinding, follow up, comparability at baseline, contamination avoided: 1/30 (0.3%) Way of using the quality assessment results declared in the method section: ? Not specified: 13/30: (43.3%) ? As inclusion criteria: 8/ 30 (26.6%) ? For sensitivity analysis: 6/30 (20%) ? Results to be described only: 3/30 (1%) Way to use the results declared in the methods section actually adopted in the results section: ? Declared but not adopted :1/19 (0.5%) ? Not declared in the methods section:8/19 (42.1%) ? Actually adopted: 7/19 (36.8%) ? Sensitivity analysis declared not performed because of heterogeneity of studies or small number of included studies: 3/19 (15.7%)
Conclusion: Even within a single review group, heterogenous approaches for evaluating quality of primary studies are used. The majority of the reviews use the allocation concealment or the Guideline of the Drug and Alcohol Group as quality criteria. For many reviews it is not clear how the studies are classified as A,B,C,or D except those who rely on allocation concealment only both to assess the methodological quality, and to classify the studies. About half of the reviews don t declare how quality assessment results will be used, while all but one actually adopt in the analysis the method declared. The variability observed reflects the debate in the Collaboration on this issue and highlights the need of defining a more uniform strategy. The actual use of the results of quality assessment is very poor. The development of specific quality guidelines did not help both in standardising the methods used and in applying results of the quality assessment.
Methods: We checked the 19 reviews and 11 protocols published by the Group (CLib 1.2004) to answer the following questions: ? Which criteria were inspected for the quality assessment of studies? ? How were they used to classify studies in the A,B,C,D classes? ? How the quality assessment results were meant to be used in the review (i.e. for inclusion criteria, for sensitivity analysis) or only described? ? If intention to use results of quality assessment is actually adopted in the results section. The first three questions were applied both to reviews and protocols, the last one only to reviews.
Results: Criteria used for the quality assessment: ? Not specified: 3/30 (1%) ? Guideline of the Drug and Alcohol Group:8/30 (26.6%) ? Allocation concealment only: 12/30 (40%) ? Randomization method and allocation concealment: 2/30 (0.6) ? Randomization method, allocation concealment, and follow up: 2/30 (0,6%) ? Randomization method, allocation concealment, blinding, and follow up: 3/30 (1%) Criteria used to classify the studies in the A,B,C,D classes ? Not specified: 12/30 (40%) ? Allocation concealment: 14/30 (46.6%) ? Randomization method, allocation concealment: 1/30 (0.3%) ? Randomization method, allocation concealment, follow up: 2/30 (0,6%) ? Randomization method, allocation concealment, blinding, follow up, comparability at baseline, contamination avoided: 1/30 (0.3%) Way of using the quality assessment results declared in the method section: ? Not specified: 13/30: (43.3%) ? As inclusion criteria: 8/ 30 (26.6%) ? For sensitivity analysis: 6/30 (20%) ? Results to be described only: 3/30 (1%) Way to use the results declared in the methods section actually adopted in the results section: ? Declared but not adopted :1/19 (0.5%) ? Not declared in the methods section:8/19 (42.1%) ? Actually adopted: 7/19 (36.8%) ? Sensitivity analysis declared not performed because of heterogeneity of studies or small number of included studies: 3/19 (15.7%)
Conclusion: Even within a single review group, heterogenous approaches for evaluating quality of primary studies are used. The majority of the reviews use the allocation concealment or the Guideline of the Drug and Alcohol Group as quality criteria. For many reviews it is not clear how the studies are classified as A,B,C,or D except those who rely on allocation concealment only both to assess the methodological quality, and to classify the studies. About half of the reviews don t declare how quality assessment results will be used, while all but one actually adopt in the analysis the method declared. The variability observed reflects the debate in the Collaboration on this issue and highlights the need of defining a more uniform strategy. The actual use of the results of quality assessment is very poor. The development of specific quality guidelines did not help both in standardising the methods used and in applying results of the quality assessment.