Article type
Year
Abstract
Background: The evaluation of imprecision is a key dimension of the 'Risk of bias' assessment. GRADE gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use of it. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision.
Objectives: To evaluate reporting of and adherence to GRADE, and to compare the assessment of imprecision of intervention effects assessed by GRADE and Trial Sequential Analysis (TSA) in Cochrane systematic reviews.
Methods: A cross-sectional study: we included 100 Cochrane Reviews irrespective of the type of intervention, provided they considered a key dichotomous outcome meta-analysed and assessed by GRADE. We assessed the Methods and Results sections of each review for adequacy of evaluation of imprecision. We reanalysed imprecision following the GRADE Handbook and the TSA Manual.
Results: Overall, only 12.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of confidence intervals (CI) and the optimal information size (Table 1). Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was reanalysed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors’ assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% CI 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors on GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between GRADE reanalysed following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79; Table 2).
Conclusions: In a sample of Cochrane Reviews, methods for assessing imprecision were rarely reported. Quality of evidence was often downgraded due to imprecision. Strict adherence to GRADE Handbook guidelines led to more frequent downgrading because of imprecision, similar to that obtained by TSA.
Patient or healthcare consumer involvement: Our project is a step towards the need for clinicians and researchers to find the most accurate method to be confident in the clinical findings so that reliable and precise results can be transferred safely to clinical practice.
Objectives: To evaluate reporting of and adherence to GRADE, and to compare the assessment of imprecision of intervention effects assessed by GRADE and Trial Sequential Analysis (TSA) in Cochrane systematic reviews.
Methods: A cross-sectional study: we included 100 Cochrane Reviews irrespective of the type of intervention, provided they considered a key dichotomous outcome meta-analysed and assessed by GRADE. We assessed the Methods and Results sections of each review for adequacy of evaluation of imprecision. We reanalysed imprecision following the GRADE Handbook and the TSA Manual.
Results: Overall, only 12.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of confidence intervals (CI) and the optimal information size (Table 1). Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was reanalysed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors’ assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% CI 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors on GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between GRADE reanalysed following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79; Table 2).
Conclusions: In a sample of Cochrane Reviews, methods for assessing imprecision were rarely reported. Quality of evidence was often downgraded due to imprecision. Strict adherence to GRADE Handbook guidelines led to more frequent downgrading because of imprecision, similar to that obtained by TSA.
Patient or healthcare consumer involvement: Our project is a step towards the need for clinicians and researchers to find the most accurate method to be confident in the clinical findings so that reliable and precise results can be transferred safely to clinical practice.