Article type
Year
Abstract
Background: The published clinical research literature may be distorted by the pursuit of statistically significant results.
Objectives: We aimed to develop a unifying concept for biases stemming from the pursuit of nominal significance and develop a test to explore these biases.
Methods: We use the term significance-chasing bias to denote the composite of three biases: lack of publication of studies with non-significant results (publication bias); selection of analysis and outcome definitions in a way that otherwise non-significant results cross the threshold of statistical significance (typically p<0.05); and fabrication of statistically significant data. All three components lead to a relative excess of formally significant findings in the published literature. Based on this premise, we have developed an exploratory test that estimates the number of expected studies with statistically significant results and compares it against the number of observed significant studies. The main application uses alpha=0.05, but a range of alpha thresholds is also examined. Different plausible values of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).
Results: We evaluated illustratively eight meta-analyses of clinical trials with >50 studies each and 10 meta-analyses of clinical efficacy for neuroleptic agents in schizophrenia; the 10 meta-analyses were also examined as a composite domain. Different results were obtained against commonly used tests of publication bias. We demonstrated a clear or possible excess of significant studies in six of eight large meta-analyses and in the wide domain of neuroleptic treatments.
Conclusions: Researchers should be made aware of the potential impact of significance-chasing bias in clinical research that goes beyond simple publication bias.
Objectives: We aimed to develop a unifying concept for biases stemming from the pursuit of nominal significance and develop a test to explore these biases.
Methods: We use the term significance-chasing bias to denote the composite of three biases: lack of publication of studies with non-significant results (publication bias); selection of analysis and outcome definitions in a way that otherwise non-significant results cross the threshold of statistical significance (typically p<0.05); and fabrication of statistically significant data. All three components lead to a relative excess of formally significant findings in the published literature. Based on this premise, we have developed an exploratory test that estimates the number of expected studies with statistically significant results and compares it against the number of observed significant studies. The main application uses alpha=0.05, but a range of alpha thresholds is also examined. Different plausible values of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).
Results: We evaluated illustratively eight meta-analyses of clinical trials with >50 studies each and 10 meta-analyses of clinical efficacy for neuroleptic agents in schizophrenia; the 10 meta-analyses were also examined as a composite domain. Different results were obtained against commonly used tests of publication bias. We demonstrated a clear or possible excess of significant studies in six of eight large meta-analyses and in the wide domain of neuroleptic treatments.
Conclusions: Researchers should be made aware of the potential impact of significance-chasing bias in clinical research that goes beyond simple publication bias.