Article type
Year
Abstract
Objectives:
It has been suggested that contradictions and initial overestimates are not unusual among highly cited studies of clinical interventions in medicine. However, this issue has not been researched in psychiatry. We investigated how highly cited studies in the field of psychiatry are replicated by subsequent studies.
Methods:
We selected highly cited studies claiming to describe effective psychiatric treatments published in five major psychiatry journals and three general medicine journals in the years 2000 through 2002. For each of these studies, we searched for subsequent studies with a better-controlled design or a similar design, but with a bigger sample size, that examined the same clinical question and compared the standardized mean differences (SMDs).
Results:
Among 83 articles recommending certain psychiatric interventions, 40 (48%) remained unchallenged in the ensuing decade; of the remaining 43 articles with replication studies, 16 (37%) were contradicted, 11 (26%) were confirmed, but were found to have substantively smaller effects, and only 16 (37%) were replicated. The SMDs of the initial studies were overestimated by 132% in comparison with the subsequent studies. Studies with a total sample size of 100 or greater tended to produce replicable results.
Conclusions:
Clinicians should be more judicious when they read research studies in the field of psychiatry, even if they are published in high impact journals and are cited frequently. Even more caution is needed when a study with a small sample size reports a big effect.
It has been suggested that contradictions and initial overestimates are not unusual among highly cited studies of clinical interventions in medicine. However, this issue has not been researched in psychiatry. We investigated how highly cited studies in the field of psychiatry are replicated by subsequent studies.
Methods:
We selected highly cited studies claiming to describe effective psychiatric treatments published in five major psychiatry journals and three general medicine journals in the years 2000 through 2002. For each of these studies, we searched for subsequent studies with a better-controlled design or a similar design, but with a bigger sample size, that examined the same clinical question and compared the standardized mean differences (SMDs).
Results:
Among 83 articles recommending certain psychiatric interventions, 40 (48%) remained unchallenged in the ensuing decade; of the remaining 43 articles with replication studies, 16 (37%) were contradicted, 11 (26%) were confirmed, but were found to have substantively smaller effects, and only 16 (37%) were replicated. The SMDs of the initial studies were overestimated by 132% in comparison with the subsequent studies. Studies with a total sample size of 100 or greater tended to produce replicable results.
Conclusions:
Clinicians should be more judicious when they read research studies in the field of psychiatry, even if they are published in high impact journals and are cited frequently. Even more caution is needed when a study with a small sample size reports a big effect.