Article type
Year
Abstract
Background: Meta-epidemiological studies have reported empirical evidence of associations between trial characteristics and intervention effect estimates. We previously suggested using such data to provide prior estimates of the bias associated with trial characteristics and hence a rational basis to summarise all relevant evidence in meta-analyses.
Objectives: To examine the implications of different assumptions about the nature of the bias associated with trial characteristics for the information contributed to meta-analyses by lower quality trials.
Methods: We considered meta-analyses containing two trial types: H (high quality, e.g. adequately concealed) and L (low quality, e.g. inadequately concealed). We formulated prior distributions for the bias associated with type L studies. Formulae for the contribution of type L studies to meta-analyses were derived. Parameters of prior distributions were estimated using the data of Schulz et al1 and WinBUGS software.
Results: Three types of uncertainty contribute to down-weighting of evidence from type L studies: (1) between-trial variability in the bias, (2) uncertainty in the estimated average bias and (3) between-meta-analysis variability in the average bias. These have different implications for the information that type L studies can contribute to meta-analyses. A large number of large type L studies can overcome uncertainty due to (1), while conducting more empirical research can reduce (2). However, down-weighting due to (3) cannot be overcome by collecting more type L data. Standard sensitivity analyses correspond to simplifications of this approach making implausibly strong assumptions about the amount of bias. Applications will be presented.
Conclusions: Meta-analyses including low quality studies and analysed using standard methods will produce biased and over-precise intervention effect estimates. Low quality studies can contribute to intervention effect estimates without introducing bias, based on prior information derived from empirical studies. Their contribution will however be limited, unless they are large and there are many of them.
References
1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-12.
Objectives: To examine the implications of different assumptions about the nature of the bias associated with trial characteristics for the information contributed to meta-analyses by lower quality trials.
Methods: We considered meta-analyses containing two trial types: H (high quality, e.g. adequately concealed) and L (low quality, e.g. inadequately concealed). We formulated prior distributions for the bias associated with type L studies. Formulae for the contribution of type L studies to meta-analyses were derived. Parameters of prior distributions were estimated using the data of Schulz et al1 and WinBUGS software.
Results: Three types of uncertainty contribute to down-weighting of evidence from type L studies: (1) between-trial variability in the bias, (2) uncertainty in the estimated average bias and (3) between-meta-analysis variability in the average bias. These have different implications for the information that type L studies can contribute to meta-analyses. A large number of large type L studies can overcome uncertainty due to (1), while conducting more empirical research can reduce (2). However, down-weighting due to (3) cannot be overcome by collecting more type L data. Standard sensitivity analyses correspond to simplifications of this approach making implausibly strong assumptions about the amount of bias. Applications will be presented.
Conclusions: Meta-analyses including low quality studies and analysed using standard methods will produce biased and over-precise intervention effect estimates. Low quality studies can contribute to intervention effect estimates without introducing bias, based on prior information derived from empirical studies. Their contribution will however be limited, unless they are large and there are many of them.
References
1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-12.