Article type
Year
Abstract
Background: stratification by quality (known as sensitivity analyses or exploratory subgroup analyses) is a commonly advocated analytical strategy, after 'Risk of bias' assessment. However, this process shuns stratification by quality score in favour of stratification by domain judgements, following research that reported inconsistency in the effect estimates between high- and low-quality trials across 25 quality scales.
Objectives: we examined if restricting primary meta-analyses to studies at low risk of bias or presenting meta-analyses stratified according to risk of bias is indeed the right approach to explore potential methodological bias.
Methods: re-analysis of the impact of quality subgroupings in an existing meta-analysis based on 25 different scales.
Results: we demonstrate that quality stratification itself is the problem because it induces a spurious association between effect size and precision within stratum. Studies with larger effects or lesser precision tend to be of lower quality, a form of collider-stratification bias (stratum being the common effect of the reasons for these two outcomes) that leads to inconsistent results across scales. We also show that the extent of this association determines the variability in effect size and statistical significance across strata when conditioning on quality.
Conclusions: we conclude that stratification by quality leads to a form of selection bias (collider-stratification bias) and should be avoided. We demonstrate consistent results with an alternative method that includes all studies.
Patient or healthcare consumer involvement: clinicians and their patients require a summary of available evidence after risk of bias has been considered. While identifying research studies at risk of being biased in meta-analysis is now mandatory (Cochrane guidelines), there is no consensus on how to use these judgments to 'adjust' research synthesis results to reduce possible bias. One such method that is commonly used, stratification by quality, has been shown in this research to induce a selection bias and therefore may not be an appropriate method. This information is really important for clinicians and their patients to accurately assess the trustworthiness of meta-analyses. This research demonstrates pitfalls in stratification method for bias adjustment in research synthesis thereby alerting researchers and thereby creating better outcomes for patients and consumers.
Objectives: we examined if restricting primary meta-analyses to studies at low risk of bias or presenting meta-analyses stratified according to risk of bias is indeed the right approach to explore potential methodological bias.
Methods: re-analysis of the impact of quality subgroupings in an existing meta-analysis based on 25 different scales.
Results: we demonstrate that quality stratification itself is the problem because it induces a spurious association between effect size and precision within stratum. Studies with larger effects or lesser precision tend to be of lower quality, a form of collider-stratification bias (stratum being the common effect of the reasons for these two outcomes) that leads to inconsistent results across scales. We also show that the extent of this association determines the variability in effect size and statistical significance across strata when conditioning on quality.
Conclusions: we conclude that stratification by quality leads to a form of selection bias (collider-stratification bias) and should be avoided. We demonstrate consistent results with an alternative method that includes all studies.
Patient or healthcare consumer involvement: clinicians and their patients require a summary of available evidence after risk of bias has been considered. While identifying research studies at risk of being biased in meta-analysis is now mandatory (Cochrane guidelines), there is no consensus on how to use these judgments to 'adjust' research synthesis results to reduce possible bias. One such method that is commonly used, stratification by quality, has been shown in this research to induce a selection bias and therefore may not be an appropriate method. This information is really important for clinicians and their patients to accurately assess the trustworthiness of meta-analyses. This research demonstrates pitfalls in stratification method for bias adjustment in research synthesis thereby alerting researchers and thereby creating better outcomes for patients and consumers.