Article type
Year
Abstract
Background: Several meta-epidemiological studies have been published that examine the relationship between attributes of study design and bias. The findings from these studies show some variation, but generally favor common assumptions of smaller effect sizes for rigorous design.
Objectives: The goal of this systematic review of meta-epidemiological studies was to evaluate the quality of the literature describing the impact of study design on the validity of study results and to assess the agreement between stated conclusions and the data provided.
Methods: A search of Medline, Embase, Web of Science, Cochrane methodology register, and reference lists for meta-epidemiological studies was conducted. References were selected by predefined criteria and appraised with an adjusted Amstar tool. We examined the following study characteristics or quality measures: randomization, allocation concealment, blinding, study size, and sponsoring. The directions of conclusions were categorized as larger effect, smaller effect, or no effect from the presence of the quality measure. Effect sizes and significance or confidence intervals were also extracted. All steps were performed in duplicate. The agreement between available data and the direction of conclusions were analyzed with Chi2 test.
Results: We found 43 meta-epidemiological studies that together stated 63 conclusions about the effect of one or more of the defined quality measures. From 22 of these studies we were able to extract 59 effect estimates of the quality measures. Eleven of these were statistically significant. Only 28/63 or 44% of conclusions were supported by the data. A conclusion of no effect was more often supported by data than a conclusion for a smaller effect from the presence of a quality measure (p < 0.01).
Conclusions: Most of the conclusions in meta-epidemiological studies are not properly supported by data, especially in cases where conclusions supported the common paradigm of smaller effects in studies that apply generally accepted quality measures.
Objectives: The goal of this systematic review of meta-epidemiological studies was to evaluate the quality of the literature describing the impact of study design on the validity of study results and to assess the agreement between stated conclusions and the data provided.
Methods: A search of Medline, Embase, Web of Science, Cochrane methodology register, and reference lists for meta-epidemiological studies was conducted. References were selected by predefined criteria and appraised with an adjusted Amstar tool. We examined the following study characteristics or quality measures: randomization, allocation concealment, blinding, study size, and sponsoring. The directions of conclusions were categorized as larger effect, smaller effect, or no effect from the presence of the quality measure. Effect sizes and significance or confidence intervals were also extracted. All steps were performed in duplicate. The agreement between available data and the direction of conclusions were analyzed with Chi2 test.
Results: We found 43 meta-epidemiological studies that together stated 63 conclusions about the effect of one or more of the defined quality measures. From 22 of these studies we were able to extract 59 effect estimates of the quality measures. Eleven of these were statistically significant. Only 28/63 or 44% of conclusions were supported by the data. A conclusion of no effect was more often supported by data than a conclusion for a smaller effect from the presence of a quality measure (p < 0.01).
Conclusions: Most of the conclusions in meta-epidemiological studies are not properly supported by data, especially in cases where conclusions supported the common paradigm of smaller effects in studies that apply generally accepted quality measures.